Koninklijke Bibliotheek, National Library of the Netherlands
IP1508762898432
Equivalence testing using existing reference data: An example with genetically modified and conventional crops in animal feeding studies
van der Voet, Hilko
Goedhart, Paul W.
Schmidt, Kerstin
text
article
monographic
Food and chemical toxicology
continuing
02786915
0000000010573
109
P1
text
Born digital tijdschriften
KB collectiekavel
text
Elektronische Wetenschappelijke Tijdschriften
EWTIJ
10.1016/j.fct.2017.09.044
urn:nbn:nl:kb-1508762898176
Automatisch gegenereerd op basis van de EWTIJ XML in release 1.5 van het digitaal magazijn.
FCT
9310
S0278-6915(17)30563-X
10.1016/j.fct.2017.09.044
The Authors
Fig. 1
Power of equivalence test in simulations (1000 runs) vs. a background in historical studies with similar design as that of most variables in the A, B and C studies (8 replicates in A and B, 5 replicates in C). Power is shown as a function of the true effect size between T and C (expressed in SD). (a) results for three values of the true variation between reference foods (
σ
R
/
σ
E
=
0,0.5
,
1
),
n
T
=
n
C
=
8
replications; (b) results for larger sample sizes in the current study,
n
T
=
n
C
=
8,16,32,64,128
replications, for
σ
R
/
σ
E
=
0
; (c) results when using additional groups to have more degrees of freedom in the current study,
d
f
F
=
14,35,200
, for
n
T
=
n
C
=
8
replications and
σ
R
/
σ
E
=
0
.
Fig. 1
Fig. 2
Confidence intervals Test vs. Control in study D, Males, compared to the non-GM background in studies ABC (12 outliers excluded), requiring
n
0
=
8
as the minimum number of cages per group. Shown are the observed ratios corresponding to differences
D
on the log scale, with 95% confidence limits (black symbols and line segments), the median equivalence limits EL (red bars delimiting a green equivalence band width), and the 2.5% and 97.5% confidence limits for EL (blue bars). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
Fig. 2
Fig. 3
Equivalence testing Test vs. Control in study D, Males, compared to the non-GM background in studies ABC (12 outliers excluded), requiring
n
0
=
8
as the minimum number of cages per group. Median ELSD with 92.5–95% confidence interval (see text).
Fig. 3
Fig. 4
Equivalence testing Test vs. Control in study D, Males, compared to the non-GM background in studies ABC, requiring
n
0
=
5
as the minimum number of cages per group. Median ELSD with 92.5–95% confidence interval (see text). Results shown before (upper, black) and after (lower, red) excluding 12 outliers from the historical data (two outliers for Liver and ALT, one outlier for growthR, Pancreas, HGB, MCH, MCHC, AST, Alb and P). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
Fig. 4
Fig. 5
Equivalence testing Test vs. Control in study D, Males, compared to the non-GM background in studies ABC (12 outliers excluded), requiring
n
0
=
8
as the minimum number of cages per group. Results for the standard model (1) (upper, black) and simplified model (6) with
σ
R
2
=
0
(lower, red). Median ELSD with 92.5–95% confidence interval (see text). (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)
Fig. 5
Table 1
Six equivalence criteria for comparing a Test (T) type to either a population of Reference (R) types or a single Control (C) type. The entry for DWE comparing T and C (
θ
) is the criterion used in this paper.
Table 1
Criterion
Compare T to R
Compare T to C
Average equivalence (AE)
(
μ
T
−
μ
R
)
2
(
μ
T
−
μ
C
)
2
Scaled average equivalence (SAE)
(
μ
T
−
μ
R
)
2
E
(
R
i
1
−
R
i
2
)
2
=
(
μ
T
−
μ
R
)
2
2
σ
R
2
(
μ
T
−
μ
C
)
2
E
(
R
i
1
−
R
i
2
)
2
=
(
μ
T
−
μ
C
)
2
2
σ
R
2
Distribution-wise equivalence (DWE)
i
,
i
1
and
i
2
represent reference feeds in the historic study
E
(
y
T
j
k
−
y
i
j
k
)
2
E
(
y
i
1
j
k
1
−
y
i
2
j
k
2
)
2
=
=
(
μ
T
−
μ
R
)
2
+
σ
R
2
+
σ
E
2
+
σ
F
2
2
σ
R
2
+
2
σ
E
2
θ
=
E
(
y
Tjk
−
y
Cjk
)
2
E
(
y
i
1
j
k
1
−
y
i
2
j
k
2
)
2
=
=
(
μ
T
−
μ
C
)
2
+
2
σ
F
2
2
σ
R
2
+
2
σ
E
2
Table 2
Summary statistics Study D against background from studies A-C. Male rats, outliers excluded.
D
= difference Test (GM, 33%) and Control in current study,
s
F
is current-study within-group standard deviation with
d
f
F
degrees of freedom,
s
E
and
s
R
are historical-study within- and between-group standard deviations with
d
f
E
and
d
f
R
degrees of freedom,
n
eff
is effective sample size in the historical studies. All statistics are calculated under model (1), except
s
E
in the last column, which is calculated under model (6).
Table 2
study
D (GM vs. Control)
A, B, C (non-GM)
variable
D
s
F
d
f
F
s
E
d
f
E
s
R
d
f
R
n
eff
s
R
2
+
s
E
2
s
E
(M6)
BodyWeight
0.012
0.052
8
0.056
61
0.000
4
10.5
0.056
0.055
growthR
0.000
0.019
8
0.019
60
0.000
4
10.3
0.019
0.019
Kidney
−0.041
0.054
8
0.054
42
0.000
4
8.0
0.054
0.053
Spleen
0.070
0.080
8
0.086
42
0.000
4
8.0
0.086
0.086
Liver
0.028
0.049
8
0.051
40
0.006
4
7.7
0.051
0.051
AdrenGl
0.026
0.085
8
0.120
42
0.000
4
8.0
0.120
0.119
Lung
−0.041
0.064
8
0.077
42
0.035
4
8.0
0.085
0.083
Heart
0.007
0.035
8
0.051
42
0.013
4
8.0
0.052
0.052
Thymus
−0.083
0.087
8
0.184
42
0.004
4
8.0
0.184
0.184
Pancreas
0.063
0.133
8
0.145
41
0.056
4
7.8
0.156
0.153
Testis
−0.057
0.074
8
0.076
42
0.000
4
8.0
0.076
0.075
Epididymis
−0.041
0.104
8
0.081
42
0.000
4
8.0
0.081
0.079
Brain
−0.016
0.055
8
0.050
42
0.016
4
8.0
0.052
0.052
WBC
0.244
0.154
8
0.220
51
0.089
4
9.2
0.237
0.232
RBC
−0.023
0.034
8
0.053
51
0.000
4
9.2
0.053
0.052
HGB
0.011
0.028
8
0.043
50
0.006
4
9.1
0.044
0.043
HCT
−0.004
0.023
8
0.051
51
0.000
4
9.2
0.051
0.050
MCV
0.019
0.020
8
0.019
51
0.000
4
9.2
0.019
0.018
MCH
0.033
0.032
8
0.027
50
0.000
4
9.1
0.027
0.026
MCHC
0.015
0.017
8
0.016
50
0.003
4
9.1
0.016
0.016
PLT
−0.071
0.111
8
0.270
51
0.000
4
9.2
0.270
0.262
LYMcount
0.228
0.153
8
0.217
51
0.076
4
9.2
0.230
0.225
Lymphocyte
0.058
0.046
8
0.045
51
0.003
4
9.2
0.045
0.045
Neutrophil
−0.254
0.153
8
0.166
51
0.000
4
9.2
0.166
0.165
Monocyte
0.065
0.279
8
0.294
51
0.126
4
9.2
0.320
0.312
Eosinophil
0.478
0.380
8
0.457
51
0.164
4
9.2
0.486
0.476
ALP
−0.206
0.162
8
0.176
51
0.000
4
9.2
0.176
0.172
ALT
0.045
0.119
8
0.120
49
0.000
4
8.9
0.120
0.119
AST
0.171
0.159
8
0.156
50
0.000
4
9.1
0.156
0.155
Alb
−0.023
0.051
8
0.055
50
0.000
4
9.1
0.055
0.054
Glu
−0.038
0.090
8
0.146
51
0.038
4
9.2
0.150
0.149
Krea
0.000
0.126
8
0.132
51
0.029
4
9.2
0.135
0.134
TP
−0.020
0.041
8
0.040
51
0.000
4
9.2
0.040
0.040
Urea
−0.106
0.062
8
0.100
51
0.057
4
9.2
0.116
0.111
CHOL
0.043
0.120
8
0.110
51
0.054
4
9.2
0.123
0.119
Ca
0.017
0.016
8
0.064
51
0.012
4
9.2
0.065
0.065
Cl
−0.016
0.010
8
0.051
51
0.002
4
9.2
0.051
0.051
K
0.066
0.071
8
0.112
51
0.000
4
9.2
0.112
0.110
Na
−0.003
0.013
8
0.056
51
0.000
4
9.2
0.056
0.056
P
0.074
0.072
8
0.101
50
0.029
4
9.1
0.105
0.103
TAG
0.065
0.233
8
0.294
51
0.178
4
9.2
0.343
0.328
Equivalence testing using existing reference data: An example with genetically modified and conventional crops in animal feeding studies
Hilko
van der Voet
a
∗
hilko.vandervoet@wur.nl
Paul W.
Goedhart
a
paul.goedhart@wur.nl
Kerstin
Schmidt
b
kerstin.schmidt@biomath.de
a
Wageningen University & Research, Biometris, Droevendaalsesteeg 1, 6708PB Wageningen, The Netherlands
Wageningen University & Research, Biometris
Droevendaalsesteeg 1
Wageningen
6708PB
The Netherlands
b
BioMath GmbH, Friedrich-Barnewitz-Str. 8, 18119 Rostock-Warnemünde, Germany
BioMath GmbH
Friedrich-Barnewitz-Str. 8
Rostock-Warnemünde
18119
Germany
∗
Corresponding author.
Abstract
An equivalence testing method is described to assess the safety of regulated products using relevant data obtained in historical studies with assumedly safe reference products. The method is illustrated using data from a series of animal feeding studies with genetically modified and reference maize varieties. Several criteria for quantifying equivalence are discussed, and study-corrected distribution-wise equivalence is selected as being appropriate for the example case study. An equivalence test is proposed based on a high probability of declaring equivalence in a simplified situation, where there is no between-group variation, where the historical and current studies have the same residual variance, and where the current study is assumed to have a sample size as set by a regulator. The method makes use of generalized fiducial inference methods to integrate uncertainties from both the historical and the current data.
Graphical abstract
Image 1
Highlights
•
An equivalence testing method is proposed to assess the safety of regulated products.
•
We combine data from a current study with test and control, and historical studies with assumedly safe reference products.
•
The method is illustrated with animal feeding studies using genetically modified and reference maize varieties.
•
A high statistical power of the equivalence test is the basis for the equivalence criterion.
•
Generalized fiducial inference is used to integrate uncertainties from the historical and the current data.
Keywords
Food safety
Average equivalence
Distribution-wise equivalence
Linear mixed model
Generalized fiducial inference
Statistical power
1
Introduction
There are categories of innovative products that are only allowed on the market after a risk assessment has shown that the product is safe for human health and the environment. Examples of regulated products are drugs, pesticides, and, in e.g. Europe, genetically modified organisms (EFSA, 2010b, 2011a). The risk assessment generally involves a comparative trial in which the new product is compared with already established products. Assessing safety by means of such a comparative trial is fundamentally different from data analysis in other scientific fields. In most other testing situations the intention is to prove the existence of a difference or an effect, for example less incidence of a disease or a higher yield of an agricultural crop. Therefore, following the principle that the intention of testing is to prove a hypothesis to be wrong, in such studies a null hypothesis of no effect is tested against an alternative hypothesis that there is an effect. However, statistical hypothesis testing is an asymmetric procedure, and absence of a significant difference cannot be considered as a statement about the truth of the null hypothesis, or “absence of evidence is not evidence of absence” (Altman and Bland, 1995).
It is thus impossible to prove exact equality using statistical procedures. Consequently, safety assessment requires the use of alternative testing procedures, collectively known as equivalence testing, to demonstrate that an effect is small enough (Schuirmann, 1987; Walker and Nowacki, 2011). Equivalence testing is well established as the regulatory required procedure in drug testing (FDA, 2003; EMA, 2010), and has also been proposed in other fields (Garrett, 1997; Cosacov et al., 2008; van der Voet et al., 2011; EFSA, 2011a; Fessel and Snedeker, 2011; Beninger et al., 2012).
Equivalence testing raises two questions. First, on which scale should potential effects be expressed, and second, what is the threshold, also called acceptance criterion or equivalence limit, for ‘small enough’? For example, in testing the equivalence of drugs, regulatory authorities prescribe that certain pharmacokinetic parameters describing the fate of a drug in human subjects should, expressed as an average, differ less than a factor of 1.25 between the tested drug and a reference drug (FDA, 2003; EMA, 2010). This approach is called average equivalence (AE).
The optimal procedure for equivalence testing has been much debated in several fields. In drug bioequivalence testing, it has been proposed to consider the full distribution of measurements rather than an average. This leads to more refined concepts such as population bioequivalence, related to the prescribability of drugs, and individual bioequivalence, related to the switchability between drugs (Midha et al., 1999; Schall and Endrenyi, 2010). In food safety assessment, the concept of substantial equivalence was introduced (OECD, 1993; Kuiper et al., 2001), but its use was mostly in a non-statistical way and linked to a strong limitation in the number of variables that should be measured. This led to criticism of the concept (Ho and Steinbrecher, 1998; Millstone et al., 1999), although the basic concept of statistical equivalence testing based on an appropriate set of variables was never challenged.
More in general, it is clear that equivalence testing needs prior specification of the set of variables that should be measured, and of the equivalence criterion to be used. The set of variables to test should ideally be limited to those that are informative to risk assessment based on existing data and scientific information available. In this paper our focus is on the statistical procedure for equivalence testing and we will assume that the set of variables has already been established. Different classes of criteria have recently been discussed by Vahl and Kang (2016) in the context of field tests with genetically modified and reference crops. Of special interest to us are scaled average equivalence (SAE) and distribution-wise equivalence (DWE). Equivalence using SAE is assessed by comparing means over experimental units. Moreover the acceptance criterion is not a fixed value, but is scaled to some relevant measure of variation. In DWE, on the other hand, full distributions rather than means are compared.
All forms of equivalence testing need external input, in order to set a fixed equivalence limit (in AE and SAE), to obtain a scaling factor (in SAE), or to obtain a reference distribution (in DWE). In principle there are two approaches to obtain such external input. The first approach is that regulators or experts specify appropriate values. The factor of 1.25 used in drug equivalence testing (FDA, 2003; EMA, 2010) is an example of this approach. However, even experts often find it difficult to specify equivalence limits in this way. The second approach is to use additional data to generate the appropriate input.
In this paper we focus on the use of such additional data which are obtained for products that serve as a reference for the product to be tested. We distinguish between two cases. In the first case, the reference products have been measured under similar circumstances as the test product, so a direct comparison can be made. For example, in field tests, the test and reference varieties of a crop are often planted in the same experiment, and consequently previous work focussed on the comparison of a test variety with a population of reference varieties (van der Voet et al., 2011; Kang and Vahl, 2014; Vahl and Kang, 2016). In the second case, multiple studies are considered, and we cannot exclude unspecified differences between the measurements in different studies.
Here we consider animal feeding studies, in which not many feed varieties can be investigated in any single study, and there may be differences between the studies related to different experimental conditions. Consequently, the basic idea introduced in this paper is to compare the difference between a test (T) and a control (C) variety, obtained simultaneously in a current study, to the typical differences between reference (R) varieties obtained in one or more historical studies. In other words, the equivalence analysis is corrected for between-study differences, and the within-study variation between references R is used to set equivalence limits for the difference between T and C in the current study. Such an approach is in line with the traditional comparative approach in GMO risk assessment that comparison with available data on the nearest comparator, as well as with similar varieties on the market, should form the initial part of the assessment procedure (Kok and Kuiper, 2003). The variation between reference varieties is a point of departure in data-based approaches to set equivalence limits. In the simplest SAE approach of Vahl and Kang (2016) the equivalence limit is some factor times the variance between the reference varieties. Unfortunately, this prevents the ability to obtain useful equivalence limits when there is no, or very small, observed variation between the R varieties. In a DWE approach, as we will see, also the variance within the reference groups contributes to the total variation, allowing to obtain equivalence limits. Therefore DWE testing is the preferred approach when little or no variation between the R varieties is expected, as for example in biochemical and haematological measurements from animal feeding trials.
Vahl and Kang (2016) derived DWE equivalence limits based on a limit case where the variation between the reference varieties was assumed to be much larger than the residual variation. The statistical properties of this procedure in the opposite case, i.e. for a situation with little or no reference variation, are unknown. We propose an alternative strategy based on desired performance of the test in a simplified situation with no reference variation. In short, we will define an equivalence limit as the upper (1-
β
) limit of the upper (1-
α
) confidence limit for a statistic that quantifies a comparison between two reference varieties that in reality have no difference. This approach guarantees that equivalence can be declared with probability 1-
β
when there is no difference in reality in this simplified situation. It may be noted that power for an equivalence test is defined differently than for a traditional difference test. In the latter case it is the probability of detecting a true non-zero difference, but the power of an equivalence test is defined here as the probability of concluding equivalence when in reality there is no difference.
The use of historical data can conceptually be seen as a two-step approach: first equivalence limits are derived from the historical data, and secondly these limits are employed in the equivalence test for the current data. Indeed, the use of external fixed values (such as the factor of 1.25) can also be considered as an instance of such two-step reasoning. The two-step approach is conceptually simple, and a straight-forward use of it has been proposed as a model for GMO safety assessment (EFSA, 2010a; EFSA, 2011b; van der Voet et al., 2011). However, the price paid for this simplicity is that the uncertainty in the estimates of the equivalence limits based on the reference data (step 1) is not accounted for in step 2 (Kang and Vahl, 2014; Vahl and Kang, 2016).
To allow for all uncertainties simultaneously, it is possible to derive methods that integrate both steps and explicitly define criteria which are a function of the model parameters. Among several statistical procedures, methods of generalized fiducial inference (Weerahandi, 1993; Krishnamoorthy and Mathew, 2002; Hannig et al., 2006b, 2016; E et al., 2008; Hannig, 2009; Cisewski and Hannig, 2012) have been found useful for equivalence (McNally et al., 2003; Hannig et al., 2006a; Kang and Vahl, 2014, 2016; Vahl and Kang, 2016). We will therefore construct and use such methods.
The main aims of this paper are therefore:
1.
To adapt the equivalence criteria to study-corrected testing, i.e. to base the equivalence test on the difference between a Test and a Control in the current study, rather than on the difference between a Test and a set of reference R varieties;
2.
To propose a new equivalence criterion where a full distribution rather than a mean will be compared to a relevant reference distribution, based on desirable power in a simplified situation.
3.
To illustrate the use of the proposed method with data from five animal feeding studies which included both conventional and genetically modified feed groups.
2
Data and methods
2.1
Data
Five animal feeding studies with GM and conventional maize have been performed as part of the EU GRACE project (http://www.grace-fp7.eu). Studies A, B, D and E were 90-day studies, and study C was a 1-year study (here we consider only the results obtained at 90 days). The data from the studies are available at the CADIMA website (https://www.cadima.info). These data have been analysed before as part of the GRACE project (Schmidt et al., 2015a,b, 2016, 2017; Zeljenková et al., 2014; 2016). Here we only consider the feed groups containing 33% maize, which was the high dose level in the GRACE studies.
In studies A-E there were 3, 3, 2, 1 and 1 non-GM (reference) groups, respectively, including the isogenic (control) varieties of the GM (test) varieties. All 6 non-GM varieties in studies A and B were different, whereas in studies C, D and E some of these were used again. The variability in the reference data after 90 days in all studies was summarized in Schmidt et al. (2017). Cages were used as the experimental unit in a completely randomized design. The group sample size in studies A and B was 8 cages for all groups. In study C the group sample size was 10 cages (but only 5 for most measurements), and in studies D and E the group sample size was 5 cages.
The statistical analysis presented here was performed with results of 13 haematological variables, 15 clinical chemical variables, the final body weight (at week 13) and 11 organ weights (relative to final body weight). In addition, the growth rate
r
was estimated from the weights
W
of each individual animal by fitting the following exponential growth curve against week number
t
:
W
=
W
f
i
n
a
l
−
(
W
f
i
n
a
l
−
W
i
n
i
t
i
a
l
)
r
t
All variables were transformed to the natural logarithmic scale and then averaged to the cage level. Any outliers were identified using Grubb's test at the 1% level. For variables in which outlying observations were identified, statistical analyses were performed with outliers included and excluded to check if this made major differences. Details of the data pre-processing are documented in Supplement 1. All calculations in this work were performed with the statistical program Genstat 18 (https://www.vsni.co.uk/software/genstat/), and should be easily reproducible with other statistical software such as R or SAS.
2.2
Statistical model
The basic structure of the statistical model is similar to the model used by van der Voet et al. (2011), Kang and Vahl (2014) and Vahl and Kang (2016). Let
y
i
j
k
be the log-transformed response of feed
i
in study
j
for unit (cage)
k
. The following linear mixed model can then be used for the historical studies
j
=
1
…
n
S
with reference feeds
i
=
1
…
n
R
, and the current study with test feed
i
=
T
and control feed
i
=
C
:
(1)
y
i
j
k
=
{
μ
R
+
R
i
+
S
j
+
E
i
j
k
μ
T
+
F
T
k
μ
C
+
F
C
k
i
=
1
…
n
R
i
=
T
i
=
C
j
=
1
…
n
S
k
=
1
…
n
i
j
k
=
1
…
n
T
k
=
1
…
n
C
Parameters
μ
R
,
μ
T
and
μ
C
correspond to the expected means for the population of reference feeds R, the test feed
T
and the control feed
C
respectively. The random effect
R
i
denotes deviations from
μ
R
for the reference feeds in the historic studies and is assumed to follow a normal distribution with mean zero and variance
σ
R
2
. Study effects
S
j
, in historical studies
j
=
1
…
n
S
, are considered to be fixed. This implies that the random effects
R
i
represent variation between reference feeds within studies. The residual random effects
E
i
j
k
, for unit
k
=
1
…
n
i
j
in the historical studies, and
F
i
k
, for units
k
=
1
…
n
i
in the current study with
i
=
T
,
C
, are assumed to follow a normal distribution with mean zero and variances
σ
E
2
and
σ
F
2
respectively. The residual variances in the historic and current studies are thus allowed to be different (see Discussion for alternative possibilities). For specific experimental designs this model can be easily extended with terms for e.g. blocks within studies. Such extensions are not needed for our illustrative data.
Note that in this model there is no formal link between the model for the historical studies and the model for the current study since the two models have no parameters in common. This is the main difference with the models used by van der Voet et al. (2011), Kang and Vahl (2014) and Vahl and Kang (2016), because these were developed for experiments where reference, test and control feeds are simultaneously compared in the same experiments.
2.2.1
Statistical model for historical data
The fixed study effects
S
j
in the historical studies are considered to be nuisance parameters, and consequently ANOVA mean squares according to Henderson method III can be employed to estimate the variance parameters
σ
R
2
and
σ
E
2
(Searle et al., 1992, p. 213). Define
S
S
S
as the sums of squares due to differences between studies,
S
S
R
|
S
as the sums of squares due to differences between feeds within studies, and
S
S
E
as the residual sums of squares.
Writing the linear model for the historical data in obvious matrix notation as
y
=
X
S
a
+
X
R
b
+
e
, the accompanying degrees of freedom are given by
d
f
S
=
r
[
X
S
]
,
d
f
R
|
S
=
r
[
X
S
X
R
]
−
r
[
X
S
]
and
d
f
E
=
N
−
r
[
X
S
X
R
]
, where
r
[
·
]
is the rank of a matrix and
N
=
Σ
i
Σ
j
n
i
j
is the total number of units. The sums of squares and degrees of freedom are easily obtained by fitting the linear model. The mean squares
m
s
E
and
m
s
R
|
S
, according to Henderson method III for a two-way crossed mixed model (Searle et al., 1992, p. 213), can then be used to estimate the variance components, i.e.
σ
ˆ
E
2
=
m
s
E
=
S
S
E
/
d
f
E
and
σ
ˆ
R
2
=
(
m
s
R
|
S
−
m
s
E
)
/
n
e
f
f
, in which
n
e
f
f
=
h
7
/
d
f
R
|
S
,
h
7
=
N
−
∑
j
(
∑
i
n
i
j
2
/
N
j
)
and
N
j
=
∑
i
n
i
j
(see Searle et al., 1992, p. 211). Note that
n
e
f
f
can be interpreted as the effective unit replication; it equals the common sample size when all
n
i
j
are the same. The estimate
σ
ˆ
R
2
can become negative in which case zero will be used as an estimate. The sums of squares
S
S
R
|
S
and
S
S
E
are independent under normality, even in the unbalanced case (as shown by Searle et al., 1992, p. 73 for a one-way lay-out). Moreover
S
S
E
follows a scaled chi-squared distribution with
d
f
E
degrees of freedom, i.e.
S
S
E
∼
σ
E
2
χ
d
f
E
2
. For balanced data, i.e. all reference feeds are present in every study and the replication
n
i
j
=
n
is constant,
S
S
R
|
S
also follows a scaled chi-squared distribution now with
d
f
R
|
S
degrees of freedom:
S
S
R
|
S
∼
(
n
σ
R
2
+
σ
E
2
)
χ
d
f
R
|
S
2
. For unbalanced data with
σ
R
2
>
0
there is no chi-squared distribution associated with
S
S
R
|
S
. However the distribution of
S
S
R
|
S
can be approximated for moderately unbalanced data by means of
(
n
e
f
f
σ
R
2
+
σ
E
2
)
χ
d
f
R
|
S
2
; note that the expectation of the two distributions are equal. The quality of the approximation for the illustrative dataset is investigated in Supplement 2.
2.2.2
Statistical model for current data
The current study is conducted to assess equivalence of the test feed T and the control feed C. Note that C represents a control type with special status, and is measured in the same study as T. For example, if T is a feed with a GM variety, C could be a feed with the near-isogenic variety. The parameter of interest is therefore the expected difference between the means
Δ
=
μ
T
−
μ
C
. This is estimated by the difference between the observed means in the current study
D
=
y
T
.
−
y
C
.
, which is distributed as
D
∼
N
(
Δ
,
a
2
σ
F
2
)
, or equivalently
(
D
−
Δ
)
/
(
a
σ
F
)
∼
N
(
0
,
1
)
, in which
a
=
1
/
n
T
+
1
/
n
C
. The variance
σ
F
2
can be estimated by means of the ratio of the sums of squares
S
S
F
and the corresponding degrees of freedom
d
f
F
=
n
T
+
n
C
−
2
, and
S
S
F
follows a scaled chi-squared distribution:
S
S
F
∼
σ
F
2
χ
d
f
F
2
independently of
D
.
In summary the data obtained in the historical and current studies can be summarized by means of the statistics
S
S
R
|
S
,
S
S
E
,
D
and
S
S
F
which are mutually independent and have distributions of known form.
2.3
Equivalence criteria
In Table 1
six equivalence criteria are presented, organised in three rows representing the type of equivalence (AE, SAE or DWE) and two columns representing whether the mean
μ
T
of the test T is compared to the mean of the reference feeds
μ
R
, or to the mean
μ
C
of the control feed C. All criteria are based on (expected) squared differences between means (AE and SAE) or observations (DWE). The basic idea is that the criterion value should be small enough to be able to conclude equivalence.
We now explain why we use distribution-wise equivalence in which T is compared to C for our illustrative data. First, the ‘Compare T to C’ criteria are preferred if the R data are mainly or completely obtained from previous studies and there may be large between-study differences. Under these circumstances the estimation of
(
μ
T
−
μ
R
)
2
will be imprecise because it will include between study effects. Note that in other types of experiments, such as field trials with several plant genotypes, the ‘Compare T to R’ approach may be feasible and even preferable, especially when T, C and R groups are simultaneously compared in the same study or studies. This was in fact employed by Vahl and Kang (2016), and their criteria labelled ‘SAE-S’ and ‘DWE-C’ are simple rescalings of criteria in the ‘Compare T to R’ column of Table 1.
Second, the DWE criterion is preferred over the AE and SAE criteria for our illustrative data. A criterion for AE would compare the difference in means between the T and R (or C) groups to a fixed, externally defined, value. For example, symmetric limit values, e.g. ln (0.8) and ln (1.25), can be written as a single limit, in this case
[
ln
(
1.25
)
]
2
. However, such external fixed values are often not available. Alternatively, the SAE criterion compares the squared difference to twice the variance
σ
R
2
. A common choice for an SAE equivalence limit is
z
0.975
2
, based on the reasoning that 95% of the reference means
(
μ
R
+
R
i
)
will lie in the interval
μ
R
±
z
0.975
σ
R
(van der Voet et al., 2011; Vahl and Kang, 2016), and therefore most of the differences between two reference means will fall in the interval
±
2
z
0.975
σ
R
. However, a SAE criterion based on
σ
R
2
runs into problems when the variance component
σ
R
2
is small such that estimates of this component become zero or very small. For the experimental situation considered in this paper (animal feeding studies) this occurs quite often.
The distribution-wise equivalence (DWE) criterion avoids the problems mentioned for AE and SAE. When comparing T to C it has the form, with
i
1
and
i
2
representing two reference feeds in the historic studies:
(2)
θ
=
E
(
y
T
j
k
−
y
C
j
k
)
2
E
(
y
i
1
j
k
1
−
y
i
2
j
k
2
)
2
=
(
μ
T
−
μ
C
)
2
+
2
σ
F
2
2
σ
R
2
+
2
σ
E
2
=
Δ
2
+
2
σ
F
2
2
σ
R
2
+
2
σ
E
2
The DWE criterion considers the two populations of measurements on the experimental units in their entirety, not just the mean parameters
μ
T
and
μ
C
. The numerator of
θ
is the expectation of the squared difference between a unit with the Test feed and a unit with the Control feed in the current study. This is compared to the denominator which is the expectation of the squared difference between a unit with a reference feed and a unit with another reference feed both in the same (historical) study. Large values of
θ
may indicate lack of equivalence. Note that the additions
+
2
σ
F
2
in the numerator and
+
2
σ
E
2
in the denominator are the only differences between the DWE and the SAE criterion. The reasons to prefer DWE over SAE for data similar to our example are further discussed in the Discussion section.
2.4
Interval estimation using generalized fiducial inference
An estimate of
θ
is readily obtained by plugging in the estimates defined in the previous paragraphs for the parameters
Δ
,
σ
F
2
,
σ
R
2
and
σ
E
2
. These estimates are based on the summary statistics
D
and
S
S
F
for the current study and
S
S
R
|
S
and
S
S
E
for the historic studies. Based on these summary statistics
Y
=
(
D
,
S
S
F
,
S
S
R
|
S
,
S
S
E
)
confidence limits for
θ
can be obtained using Generalized Fiducial Inference (GFI), a theory that has been developed recently on the basis of Fisher's fiducial argument (see e.g. the review in Hannig et al., 2016). The general idea of GFI is that a Generalized Fiducial Distribution (GFD) for the unknown parameter, here
θ
, is constructed by inverting the data-generating equation. The data-generating equation is the model (1) extended with the calculations leading to the summary statistics
Y
. In a more general notation it can be specified as
Y
=
G
(
U
,
ξ
)
, for data
Y
, a deterministic function
G
(
·
,
·
)
, fixed but unknown parameters
ξ
and random components
U
with completely known distributions. If it is possible to invert the equation to
ξ
=
Q
(
U
,
Y
)
, where
Q
(
·
,
·
)
are Generalized Pivotal Quantities (GPQs), then simulating a large number (in this paper 10 000) of realizations for the random components
U
produces an empirical distribution representing the GFD for the unknown parameters
ξ
. Confidence limits can be obtained as empirical percentiles of the GFD.
As a simple example, the data generating equation for the sums of squares
S
S
F
can be written as
S
S
F
=
σ
F
2
U
F
where
U
F
is a random value from a chi-squared distribution:
U
F
∼
χ
d
f
F
2
. Inverting this gives
G
P
Q
(
σ
F
2
)
=
S
S
F
/
U
F
as the Generalized Pivotal Quantity for the parameter
σ
F
2
. Given an observed
S
S
F
,
G
P
Q
(
σ
F
2
)
can be used to derive a confidence interval for the parameter
σ
F
2
. In this simple example this can be done exactly by employing percentiles of the
χ
d
f
F
2
distribution, giving the classical confidence interval for
σ
F
2
. In a more general approach, multiple simulations from the appropriate distribution, in this case
χ
d
f
F
2
, can be employed to generate the Generalized Fiducial Distribution of the unknown parameter
σ
F
2
. Finally, 2.5th and 97.5th percentiles of the simulated GFD then give numerical estimates of the limits of a 95% two-sided confidence interval.
In the previous paragraph it was shown that the data can be summarized by means of the independent statistics
D
,
S
S
F
,
S
S
R
|
S
and
S
S
E
. These can be used in the following way to provide GPQs for the parameters
Δ
,
σ
F
2
,
σ
R
2
and
σ
E
2
, using inversion of the data generating equations given in the previous paragraphs, and defining
Z
∼
N
(
0,1
)
,
U
E
∼
χ
d
f
E
2
,
U
F
∼
χ
d
f
F
2
,
U
R
|
S
∼
χ
d
f
R
|
S
2
:
(3a)
G
P
Q
(
σ
F
2
)
=
S
S
F
/
U
F
(3b)
G
P
Q
(
Δ
)
=
D
+
a
Z
G
P
Q
(
σ
F
2
)
(3c)
G
P
Q
(
σ
E
2
)
=
S
S
E
/
U
E
(3d)
G
P
Q
(
σ
R
2
)
=
max
[
0
,
(
S
S
R
|
S
/
U
R
|
S
−
G
P
Q
(
σ
E
2
)
)
/
n
e
f
f
]
The maximisation in (3d) prevents negative estimates of
σ
R
2
, which is a commonly used strategy in estimating variance components. Note that percentiles of the simulated distribution
G
P
Q
(
Δ
)
are just numerical approximations of the usual
t
-distribution based confidence limits for
Δ
(see e.g. Example 1 in Hannig et al., 2016). The GPQ for
θ
in equation (2) is then given by replacing parameters by their respective GPQ's.
(4)
G
P
Q
(
θ
)
=
[
G
P
Q
(
Δ
)
]
2
+
2
G
P
Q
(
σ
F
2
)
2
G
P
Q
(
σ
R
2
)
+
2
G
P
Q
(
σ
E
2
)
So, conditional on the observed summary statistics
D
,
S
S
F
,
S
S
R
|
S
and
S
S
E
, equation (4) can be used to simulate a large number of values giving the generalized fiducial distribution of
θ
.
Large values of
θ
may indicate lack of equivalence, therefore the
100
(
1
−
α
)
% percentile of the GFD defined by (4),
θ
upp
=
P
100
(
1
−
α
)
(
G
P
Q
(
θ
)
)
, serves as an upper confidence limit for the magnitude of
Δ
(whether positive or negative) translated to the
θ
scale.
2.5
Distribution-wise equivalence test at the
θ
scale
Distribution-wise equivalence testing requires specification of a ‘safe’ case which forms the basis for equivalence. To set an equivalence limit at the chosen
θ
scale, we introduce a ‘safe’ case where there is no difference between T and C (i.e.
Δ
=
0
) and therefore values of
θ
are relatively low. For any dataset derived under this no-difference hypothesis, we define
θ
u
p
p
0
as an upper
100
(
1
−
α
)
% confidence limit for
θ
related to large values of
Δ
(whether positive or negative). Our requirement is that for simulations from the ‘safe’ case, this upper confidence limit will remain below the equivalence limit (and therefore indicate equivalence) with a pre-set power
1
−
β
. In other words, the equivalence limit will be set as the
100
(
1
−
β
)
% percentile of simulated upper confidence limits.
The ‘safe’ case should not depend on any of the current or historical data, but it should only depend on the design parameters of the historical studies. We define a simplified ‘safe’ case by making a number of assumptions:
a)
There is no difference between the test feed T and control feed C in the current study, or
Δ
=
0
(this corresponds to the idea of a power analysis for the equivalence test);
b)
There is no variability between the reference feeds in the historical studies, or
σ
R
2
=
0
;
c)
The residual variance in the historic and current studies are identical, or
σ
F
2
=
σ
E
2
;
d)
The regulator will set a minimum sample size
n
0
for the T and C groups to be compared. The idea is that the current experiment should not be allowed to be too small, which would lead to unacceptable wide equivalence bands. A value for
n
0
may be inspired by external guidance or by values in the historical data (summarized by the effective sample size
n
e
f
f
). In principle,
n
0
will be used to define the structure of the current study when calculating the equivalence limit, i.e.
a
0
=
2
/
n
0
and
d
f
0
=
2
n
0
−
2
.
Using a superscript ‘0’ to denote that the distributions are derived under these additional assumptions, the distributions of the four summary statistics are then given by
D
0
∼
N
(
0
,
a
0
2
σ
E
2
)
,
S
S
F
0
∼
σ
E
2
χ
d
f
0
2
,
S
S
E
0
∼
σ
E
2
χ
d
f
E
2
, and
S
S
R
|
S
0
∼
σ
E
2
χ
d
f
R
|
S
2
. All four distributions have the common variance parameter
σ
E
2
which will cancel out in equation (4), and without loss of generality we can set
σ
E
2
=
1
. The four distributions thus do not depend on any parameter; they only depend on the design of the historical studies, through the various sample sizes and degrees of freedom, and the regulatory
n
0
. A simulation approach is now used to find out which values of the DWE criterion
θ
can be expected in this simplified situation:
1)
Simulate the data summary statistics
D
0
,
S
S
F
0
,
S
S
E
0
and
S
S
R
0
according to the distributions given above, with
σ
E
2
=
1
;
2)
Use equation (4) to simulate the
GFD
0
of
θ
0
for this simulated dataset using a large number of samples (10 000 in this paper);
3)
Summarize the simulated distribution of
θ
0
by means of the
100
(
1
−
α
)
percentile
θ
u
p
p
0
=
P
100
(
1
−
α
)
(
G
P
Q
θ
0
)
of
G
F
D
0
;
4)
Repeat steps 1–3 many times (10 000 in this paper) to obtain the distribution, say
G
α
0
, of
θ
u
p
p
0
under the additional assumptions;
5)
Set the equivalence limit
θ
0
to the
100
(
1
−
β
)
percentile of the distribution
G
α
0
:
θ
0
=
P
100
(
1
−
β
)
(
θ
u
p
p
0
)
.
Under these assumptions we would like to reject the null hypothesis of no equivalence with a large probability, say
1
−
β
. The probability
1
−
β
is the power of the equivalence test in the simplified situation. Note that
θ
0
only depends on the design values of the historical studies and on three regulatory values,
n
0
,
α
and
β
.
Using the equivalence limit
θ
0
as calculated above, the equivalence test can be carried out for a dataset by calculating the generalized fiducial distribution given by equation (4) for the observed summary statistics of the historical and current data. Employing the
100
(
1
−
α
)
percentile
θ
u
p
p
of this distribution, the null hypothesis of no equivalence will be rejected when
θ
u
p
p
<
θ
0
.
2.6
Distribution-wise equivalence test at the equivalence limit scaled difference (ELSD) scale
The scale of the DWE criterion
θ
is not easily understood. It is therefore preferable to re-express results on a better recognizable scale. First we express results on the more familiar difference (
Δ
) scale between the test T and the control C. Note that for the illustrative data, variables were log-transformed such that differences in fact relate to ratios on the original scale. Secondly, for a full integration of uncertainties we perform an additional scaling to what we call the equivalence limit scaled difference (ELSD) scale. As the name indicates, +1 and −1 represent the equivalence limits on this scale.
First, on the
Δ
scale, the classical confidence interval can be obtained, either using a parametric calculation or using percentage points of
G
P
Q
(
Δ
)
. We now derive how the equivalence limit
θ
0
translates to the
Δ
scale. Define
A
=
1
/
2
G
P
Q
(
σ
F
2
)
and
B
=
[
G
P
Q
(
σ
R
2
)
+
G
P
Q
(
σ
E
2
)
]
/
G
P
Q
(
σ
F
2
)
then
(5)
G
P
Q
(
θ
)
=
[
G
P
Q
(
Δ
)
]
2
+
2
G
P
Q
(
σ
F
2
)
2
G
P
Q
(
σ
R
2
)
+
2
G
P
Q
(
σ
E
2
)
=
[
A
·
G
P
Q
(
Δ
)
]
2
+
1
B
For the purpose of deriving symmetric limits on the
Δ
scale, the equivalence limit
θ
0
therefore corresponds to equivalence limits
Δ
0
,
l
o
w
,
Δ
0
,
u
p
p
with distributions which are found by inverting equation (5):
E
L
l
o
w
,
E
L
u
p
p
=
G
P
Q
(
Δ
0
,
l
o
w
)
,
G
P
Q
(
Δ
0
,
u
p
p
)
=
±
(
B
θ
0
−
1
)
/
A
provided that
B
θ
0
−
1
>
0
. Note that
A
and
B
are transformed GFDs, and therefore the equivalence limits on the
Δ
scale are also GFDs. Also note that the equivalence limits
Δ
0
,
l
o
w
,
Δ
0
,
u
p
p
are symmetric around zero. The medians of the two GFDs can be used as point estimates, and using
100
α
/
2
and
100
(
1
−
α
/
2
)
percentile points, both GFDs can be represented by a confidence interval. We now have obtained three confidence intervals all based on a GFD: for
Δ
, for
Δ
0
,
l
o
w
and for
Δ
0
,
u
p
p
, respectively. It is not clear how to compare the interval for
Δ
with the intervals for the equivalence limits. Obviously, although the
Δ
scale has the advantage of familiarity, it is not the appropriate scale for a direct representation of the equivalence test.
Therefore, to obtain a scale where equivalence can be represented directly, we apply a further scaling using the (positive) upper equivalence limit. Define
G
P
Q
(
E
L
S
D
)
=
G
P
Q
(
Δ
)
G
P
Q
(
Δ
0
,
u
p
p
)
where ELSD is short for Equivalence Limit Scaled Difference. Now in this new measure all uncertainties are integrated into one distribution, and the equivalence condition is met when an appropriate confidence interval derived from
G
P
Q
(
E
L
S
D
)
lies completely within the interval (−1,1). We now describe how to construct this confidence interval.
If
B
θ
0
−
1
≤
0
, the corresponding estimates of
Δ
0
,
l
o
w
,
Δ
0
,
u
p
p
are set both to zero, and it is clear that equivalence cannot be established. In order to have a visual indication on the ELSD scale, and also to define a distribution of
G
P
Q
(
E
L
S
D
)
from which empirical confidence limits can be calculated, we set
G
P
Q
(
E
L
S
D
)
in those cases to a large negative or positive value, e.g.
B
I
G
=
2
, with sign equal to that of
G
P
Q
(
Δ
)
. The full definition of
G
P
Q
(
E
L
S
D
)
thus becomes
{
G
P
Q
(
E
L
S
D
)
=
G
P
Q
(
Δ
)
G
P
Q
(
Δ
0
,
u
p
p
)
if
B
θ
0
−
1
>
0
G
P
Q
(
E
L
S
D
)
=
−
B
I
G
if
B
θ
0
−
1
≤
0
and
G
P
Q
(
Δ
)
<
0
G
P
Q
(
E
L
S
D
)
=
+
B
I
G
if
B
θ
0
−
1
≤
0
and
G
P
Q
(
Δ
)
≥
0
The one-sided test using
G
P
Q
(
θ
)
can make no distinction between positive and negative differences. One-sided intervals on the
θ
scale therefore correspond to two-sided intervals on the ELSD scale with lower and upper confidence limits which are perforce symmetric around 0. By a simple search algorithm we therefore identify a limit
E
L
S
D
lim
such that
P
[
G
P
Q
(
E
L
S
D
)
<
−
E
L
S
D
l
i
m
]
+
P
[
G
P
Q
(
E
L
S
D
)
>
E
L
S
D
l
i
m
]
=
α
.
The ELSD scale can also be used for the difference test. Since
G
P
Q
(
Δ
)
defines the classical confidence interval for the difference
Δ
, and the sign of
G
P
Q
(
E
L
S
D
)
is always equal to that of
G
P
Q
(
Δ
)
, it follows that the classical difference test can be performed by checking whether the value zero is included in the confidence interval given by the
100
α
/
2
and
100
(
1
−
α
/
2
)
percentile points
E
L
S
D
100
α
/
2
and
E
L
S
D
100
(
1
−
α
/
2
)
of
G
P
Q
(
E
L
S
D
)
. There are thus two relevant intervals on the ELSD scale: the symmetric interval around zero for the equivalence test and the interval which corresponds to the classical difference test. For visualisation of both, we propose an interval with the most appropriate upper and lower limit, such that both the difference and equivalence test can be performed by this single interval. Depending on which of the difference test percentiles is closest to zero, we propose to plot intervals with limits
(
E
L
S
D
l
o
w
,
E
L
S
D
u
p
p
)
:
{
(
E
L
S
D
100
α
/
2
,
E
L
S
D
l
i
m
)
if
a
b
s
(
E
L
S
D
100
α
/
2
)
<
a
b
s
(
E
L
S
D
100
(
1
−
α
/
2
)
)
(
−
E
L
S
D
l
i
m
,
E
L
S
D
100
(
1
−
α
/
2
)
)
if
a
b
s
(
E
L
S
D
100
α
/
2
)
≥
a
b
s
(
E
L
S
D
100
(
1
−
α
/
2
)
)
These intervals, while covering between
100
(
1
−
3
α
/
2
)
and
100
(
1
−
α
)
% of the distribution, can then be used for equivalence and difference testing. The hypothesis of no difference is rejected in case the interval does not contain zero, while the non-equivalence hypothesis is rejected when the interval fully lies inside the interval (−1,1).
2.7
Simplified model assuming
σ
R
2
=
0
In animal feeding studies there is often little evidence of variation between reference feeding groups. If the assumption
σ
R
2
=
0
is made from the beginning, the statistical model (1) simplifies to
(6)
y
i
j
k
=
{
μ
R
+
S
j
+
E
R
j
k
μ
T
+
F
T
k
μ
C
+
F
C
k
i
=
R
i
=
T
i
=
C
j
=
1
…
n
S
k
=
1
…
n
R
j
k
=
1
…
n
T
k
=
1
…
n
C
The DWE equivalence criterion (2) reduces to
(7)
θ
=
Δ
2
+
2
σ
F
2
2
σ
E
2
The GPQ calculations described in the previous sections can be easily modified for this simplified model by omitting all terms involving
σ
R
2
.
2.8
Power analysis
For a power analysis
1000
datasets were simulated following the design of the A, B and C studies for the historical data, with 8 replications in studies A and B and 5 replication is study C, and a simple two-group comparison for the current study, with
n
T
=
n
C
=
8
replications and
d
f
F
=
14
degrees of freedom. It was assumed that these sample sizes were acceptable for regulators (
n
o
=
8
), and residual variances
σ
E
2
and
σ
F
2
were arbitrarily set equal to 1. The between reference variance
σ
R
2
was set to 0, 0.25 and 1 in different simulations. Moreover the effect of a larger replication in the current study was investigated using values of 16, 32, 64 and 128 for
n
T
and
n
C
. In a third set of simulations the degrees of freedom in the current study was varied between
d
f
F
= 14 (corresponding with two groups of 8) and 200 (corresponding with a current study where many additional feed groups would provide additional degrees of freedom to estimate the residual variation). The number of datasets used for calculation of
θ
0
was 10 000 and the number of GPQ samples was also set to 10 000. The level of significance
α
was set at 5% and the probability for an equivalence outcome in the simplified case
(
1
−
β
)
was set at 95%.
Fig. 1
shows the simulated power against the true effect size expressed as
Δ
/
σ
E
. The probability for an equivalence outcome in the simplified case, here set at
1
−
β
=
0.95
, is indeed attained in the simplified situation where
Δ
=
0
and
σ
R
2
=
0
. The power decreases for increasing true effect sizes. With 8 replications (Fig. 1a) feeds with a true difference of one standard deviation are still judged equivalent in 80% of the cases. True variability between reference feeds (
σ
R
2
=
0.25
or
1
) increases the power, and allows feeds with larger true differences to be considered equivalent. With more replications in the current study (Fig. 1b) the power curves start at higher power and are steeper. With more residual degrees of freedom in the current study (Fig. 1c) the power curves also start at higher values, but the differences are smaller in comparison to panel (b). Finally, note that the true effect size with power equal to 0.05 in panels (b) and (c) appears to be largely independent from the replication in the current study.
3
Results
As an illustration of the proposed methodology for equivalence testing, the observed differences between Test (GM at 33%) and Control fed male or female rats in study D or E of the Grace project were analysed against the background of the data for the Reference and Control (all non-GM) male rats in studies A, B and C. The method has been applied both with and without the identified outliers, and with different values of the regulatory sample size, i.e.
n
0
= 5 or 8, corresponding to the typical sample sizes in studies D/E and A/B, respectively. We assumed regulatory error rates
α
and
β
both equal to 5%. All results are presented graphically in Supplement 3. In this section we just show one example to illustrate the general points.
The summary statistics for the male rats of study D, after excluding 12 outliers (see Supplement 1), are given in Table 2
where, for a better interpretability, the sums of squares are recalculated as standard deviations (
s
=
S
S
/
d
f
). For the historical data, the variation between Reference groups
s
R
(estimated from 4 degrees of freedom) is always smaller than the variation within groups
s
E
, and for 20 out of 41 variables it is estimated as zero or near zero. For BodyWeight and growthR the effective sample size is larger because these variables are available for 10, rather than 5, cages in study C. Relative organ weights were only obtained from historical studies A and B, therefore the effective replication is smaller. Furthermore slight variations in degrees of freedom for error
d
f
E
and effective replication
n
e
f
f
are due to exclusion of outliers. In the last two columns of Table 2 we compare estimates of the standard deviations related to the denominator of the DWE criterion
θ
, i.e.
s
R
2
+
s
E
2
for model (1) and
s
E
for model (6). It can be noted that
s
R
2
+
s
E
2
is only slightly larger than
s
E
, with a maximum change of less than 5% (for TAG).
In a first example, we have assumed that the sample sizes in the historical studies (characterised by
n
e
f
f
between 7.7 and 10.5) roughly represent regulatory needs, and purely for illustration the regulatory minimum sample size was set to
n
0
=
8
(cages). It must be noted that in study D the sample sizes were smaller,
n
T
=
n
C
=
5
, so that a power of 0.95 will not be reached in the simplified situation.
In Fig. 2
the observed differences are expressed as ratios of Test vs. Control, i.e. differences at a logarithmic scale. The black line segments in Fig. 2 are just the GFD versions of the ordinary 95% confidence intervals for the differences at the log scale, and their intersection with the vertical line at ratio = 1 indicates non-significance of a traditional two-sided difference test. Neutrophils and Urea are seen to be significantly smaller in the Test group than in the Control group, and WBC and LYMcount are significantly greater. The new elements in Fig. 2 are the estimated equivalence limits
E
L
l
o
w
and
E
L
u
p
p
together with their 95% confidence bounds. The variables have been sorted within their category in order of increasing median equivalence limit. Except for MCH all interval estimates of the ratio Test/Control (the black lines) are between the median equivalence limits (the red limits), including those for Neutrophils and Urea. In several cases, however, the 95% confidence interval for the ratio and the 95% confidence interval for the equivalence limit do overlap. The
Δ
scale in Fig. 2 cannot be used to perform an equivalence test. For that purpose we consider the ELSD scale in Fig. 3
where all uncertainties have been incorporated in the
E
L
S
D
statistic. As explained in the Methods section, the confidence limit closest to 0, which can be used for the two-sided difference test, excludes
100
α
/
2
percent of the distribution, whereas the other confidence limit, to be used in the one-sided equivalence test, excludes between
100
α
/
2
and
100
α
percent of the distribution. From Fig. 3 it follows that 36 out of the 41 intervals (78%) are between the standardised equivalence limits −1 and +1. For 5 of the 41 variables the experiment with 5 cages per groups is insufficient to produce ELSD intervals that are fully in the equivalence range. Note, however, that all point estimates are still in the interval
±
1
, so in the terminology of EFSA (2010a, 2011a,b) equivalence is ‘more likely than not’.
Just for illustration of the method we have also assumed that the sample size of study D was in agreement with regulatory needs, e.g. the regulatory minimum sample size was set to
n
0
=
5
(cages). Fig. 4
shows that in this case all confidence intervals are in the interval
±
1
. Note that we are separately testing equivalence for 41 variables without any attempt to correct for the multiplicity, therefore around 5% of the intervals extending outside the
±
1
limits (around 2 out of 41) can be expected without statistically indicating a lack of equivalence.
Fig. 4 also shows the effects of including or excluding the identified outliers. In this example all 12 outliers occurred in the historical data (none in the current data), and therefore the result of excluding outliers is a more narrow equivalence bandwidth and consequently a somewhat wider ELSD interval. The conclusions of the equivalence tests are however not changed.
Finally, Fig. 5
compares the ELSD intervals derived under the standard model (1), which includes a term
R
i
for between-reference group variation, and the simplified model (6), where
σ
R
2
=
0
is assumed throughout. The differences found in the ELSD intervals were very small.
4
Discussion
A new method for safety assessment of innovative products has been proposed and was applied to existing data. The advantage of the proposed method is that it employs available data from previous studies to characterise the variation of observations on different reference groups that are assumed to be safe. The equivalence limits are derived in such a way that an experiment with two groups with no difference would give a confidence interval for the group difference that would lie in between the equivalence limits with a predetermined probability. In this respect the proposed approach falls under the concept of tolerance intervals which have similar properties (Kang and Vahl, 2014; Hong et al., 2014).
Statistical testing is an asymmetric procedure, both for difference and for equivalence testing. Note that in our procedure we attempt to demonstrate equivalence by rejecting a hypothesis of non-equivalence. Consequently a failure to do so is not a proof of non-equivalence. Further, non-equivalence of a Test group versus a Control groups does not imply a verdict about safety, but only about an observed difference which is larger than has been seen previously for the reference groups. Safety assessment is always more than just a statistical equivalence test, and needs the interpretation of the results by experts.
In equivalence testing for drugs, expert-based fixed values are commonly used for equivalence limits (FDA, 2003; EMA, 2010). When experts are able to set such limits, this is a reasonable procedure. For cases where experts have difficulties to translate their expertise to numerical values even when historical datasets on reference groups are available, the proposed procedure may be helpful as an alternative.
In the context of animal feeding studies the use of standardised effect size (SES) has been suggested by EFSA (2011b). SES is the observed difference between group means divided by one standard deviation between experimental units. The reasoning of EFSA was: ‘If experience from previous toxicity tests shows that an effect size of, say, one SD or less is of little toxicological relevance then this can be used to determine sample size in new situations’. In other words, EFSA sketches an example where one SD is considered an appropriate equivalence limit. In the absence of other externally provided equivalence limits SES has been used in previous analyses of the data from the GRACE study to provide a first step to show that equivalence testing is to be preferred over difference testing in the safety assessment of GM plants (Schmidt et al., 2015a, b, 2016; 2017; Zeljenková et al., 2014; 2016). In this paper we have proposed an alternative to the purely hypothetical assumption of EFSA that one SD would be a reasonable equivalence limit, by using historical data to specify equivalence limits.
We have assumed potentially different residual variances for the historical and the current study (
σ
E
2
and
σ
F
2
, respectively). If the variances would in fact be the same, there would be a benefit of pooling the variance estimates for the historical and current data. However, in regulatory equivalence testing, there is a potential danger in using the variances from the current experiment (typically under control of the applicant) in a role where a larger variance effectively may widen the equivalence region [ELlow, ELupp] of the type as shown in Fig. 2. This could lead to a situation where lack of precision in the current experiment would lead to easier acceptance of equivalence, which is undesirable. For this reason we chose the model such that the historical data (reviewed and accepted by the regulators) set a standard for variances and consequently equivalence limits. This still leaves the possibility of assuming a single residual variance in model (1) or (6) which uses two GPQ estimates in equation (4): a pooled estimate in the numerator and an estimate based on only the historical data in the denominator. However, in that case, a researcher who conducts a current experiment with improved precision would not gain the full benefit of this improvement. The proposed approach can be adapted to the needs of regulators in specific cases regarding the allowed role of data from the current study for estimating precision and also for estimating between-reference variation if more references are available.
We have argued that DWE is preferable over SAE when the between feed variance component (
σ
R
2
)
is zero or small. Application of SAE is clearly impossible when
σ
R
2
is zero, but in theory it could still be applied for any positive estimate. However, in our example the point estimates
s
R
were mostly below 0.1, and a difference of less than one standard deviation corresponds to a ratio lower than
e
x
p
(
0.1
)
−
1
=
10
%
at the original scale (only three estimates were higher: 0.178 for TAG, 0.164 for Eosinophils and 0.126 for Monocytes). Commonly, but not always, equivalence testing considers wider limits than 10%, and in such cases using estimates of variation with lower values seems contrary to the intentions. Perhaps more importantly, in our example the between feed variance component (
σ
R
2
)
could only be estimated with four degrees of freedom. As a consequence the estimates
s
R
in Table 2 are very imprecise. For example, 95% confidence intervals for
σ
R
using equation (3d) are 0.000–0.037 for BodyWeight (point estimate 0.000), and 0.067–0.563 for TAG (point estimate 0.178). In using SAE for equivalence testing, employing
0.5
G
P
Q
2
(
Δ
)
/
G
P
Q
2
(
σ
R
2
)
in analogy with the proposed approach for DWE, we would use the 95th percentile of the distribution of this ratio. This only makes sense when at least 95% of the GPQ values for
σ
R
2
are positive, but this was only the case for the five variables TAG (99.6%), Urea (99.3%), CHOL (98.3%), Monocyte (96.0%) and Lung (95.6%). Therefore, application of SAE is not an option for most variables in our case study. However, for four of the five remaining variables, the highly uncertain
σ
R
2
estimate still caused the upper 95% limit of the SAE criterion to attain much higher values than the SAE criterion
z
0.975
2
=
3.84
. The 95% upper limits were 3.15 (TAG), 11.07 (Urea), 14.60 (CHOL), 46.91 (Monocyte) and 55.85 (Lung), respectively, so that only for TAG equivalence would be shown by SAE. Note that the uncertainty of
σ
R
2
plays a smaller role in the DWE criterion, because
σ
E
2
is added to the denominator which, for the current data set, is typically a larger value with less uncertainty.
When estimates of
σ
R
2
are small, another option is to omit
σ
R
2
from the DWE model altogether. This is model (6), and for the case study in this paper the results were very similar to the results of model (1). Experience with more data sets would be needed to justify a general preference for making the extra assumption
σ
R
2
=
0
.
Estimating historical variation between feeds with only four degrees of freedom is not ideal, and is due to the lack of a long history of comparable data for non-GM foods in the test facility. The proposed method using model (1) will be most useful in an infrastructure, for example a routine testing laboratory, where a longer series of historical studies, each involving at least two non-GM feeding groups, are available for the characterisation of reference variation. A requirement of at least two reference groups per feeding study would be new, but is helpful for establishing the appropriate background data for equivalence testing using model (1).
It should be stressed that the analysis in this paper is just an illustration of methodology, and not a real safety assessment. As a first point, a full safety assessment would require case-by-case interaction with regulators and risk assessors regarding the choice of variables that is needed to cover the spectrum of possible unintended effects. Such interaction could suggest a sufficient set of variables based on potential and plausible pathways to harm. A second point would be a choice between external, expert-set equivalence limits and equivalence procedures based on historical data. In the latter case, a third point to be decided by regulators would be the specification of experimental effort, for example in the form of a fixed minimum sample size
n
0
, e.g. conforming OECD guidance, or by specifying that a current experiment would need to have at least the same sample size as has been used in historical experiments.
Acknowledgements
This work was carried out as part of the project Genetically modified plants Two Year Safety Testing (G-TwYST), financially supported by the Seventh Framework Programme (FP7) of the European Community for Research, Technological Development and Demonstration Activities, grant agreement No. 632165. Animal feeding studies were carried out as part of the project GMO Risk Assessment and Communication of Evidence (GRACE, FP7 grant agreement No. 311957). We thank our colleagues in the GRACE and G-TwYST projects for their support of this work.
Appendix A
Supplementary data
The following are the supplementary data related to this article:
Supplement 1 - Details of analysis
Supplement 1 - Details of analysis
Supplement 2 - Quality of Chi-squared approximation for SSR
Supplement 2 - Quality of Chi-squared approximation for SSR
Supplement 3 - Allresults
Supplement 3 - Allresults
Appendix A
Supplementary data
Supplementary data related to this article can be found at https://doi.org/10.1016/j.fct.2017.09.044.
Transparency document
Online data
Online data
Transparency document
Transparency document related to this article can be found online at https://doi.org/10.1016/j.fct.2017.09.044.
References
Altman and Bland, 1995
D.G.
Altman
J.M.
Bland
Statistics notes: absence of evidence is not evidence of absence
Br. Med. J.
311
1995
485
Beninger et al., 2012
P.G.
Beninger
I.
Boldina
S.
Katsanevakis
Strengthening statistical usage in marine ecology
J. Exp. Mar. Biol. Ecol.
426–427
2012
97
108
Cisewski and Hannig, 2012
J.
Cisewski
J.
Hannig
Generalized fiducial inference for normal linear mixed models
Ann. Stat.
40
2012
2102
2127
Cosacov et al., 2008
A.
Cosacov
J.
Nattero
A.A.
Cocucci
Variation of pollinator assemblages and pollen limitation in a locally specialized system: the oil-producing nierembergia linariifolia (solanaceae)
Ann. Bot.
102
2008
723
734
E L et al., 2008
L.
E
J.
Hannig
H.K.
Iyer
Fiducial intervals for variance components in an unbalanced two-component normal mixed linear model
J. Am. Stat. Assoc.
103
2008
854
865
EFSA, 2010a
EFSA
Statistical considerations for the safety evaluation of GMOs
EFSA J.
8
2010
1250
10.2903/j.efsa.2010.1250
EFSA, 2010b
EFSA
Guidance on the environmental risk assessment of genetically modified plants
EFSA J.
8
11
2010
1879
10.2903/j.efsa.2010.1879
EFSA, 2011a
EFSA
Scientific Opinion on Guidance for risk assessment of food and feed from genetically modified plants
EFSA J.
9
5
2011
2150
10.2903/j.efsa.2011.2150
EFSA, 2011b
EFSA
Guidance on conducting repeated-dose 90-day oral toxicity study in rodents on whole food/feed
EFSA J.
9
2011
2438
EMA, 2010
EMA
Guideline on the Investigation of Bioequivalence
Doc. Ref.: CPMP/EWP/QWP/1401/98 Rev. 1/Corr
2010
http://www.ema.europa.eu/docs/en_GB/document_library/Scientific_guideline/2010/01/WC500070039.pdf
FDA, 2003
FDA
Guidance for Industry: Bioavailability and Bioequivalence Studies for Orally Administered Drug Products — General Considerations
2003
http://www.fda.gov/ohrms/dockets/ac/03/briefing/3995B1_07_GFI-BioAvail-BioEquiv.pdf
Fessel and Snedeker, 2011
G.
Fessel
J.G.
Snedeker
Equivalent stiffness after glycosaminoglycan depletion in tendon — an ultra-structural finite element model and corresponding experiments
J. Theor. Biol.
268
2011
77
83
Garrett, 1997
K.A.
Garrett
Use of statistical tests of equivalence (bioequivalence tests) in plant pathology
Phytopathology
87
1997
372
374
Hannig, 2009
J.
Hannig
On generalized fiducial inference
Stat. Sin.
19
2009
491
544
Hannig et al., 2006a
J.
Hannig
E L
A.
Abdel-Karim
H.
Iyer
Simultaneous fiducial generalized confidence intervals for ratios of means of lognormal distributions
Austrian J. Stat.
35
2006
261
269
Hannig et al., 2006b
J.
Hannig
H.
Iyer
P.
Patterson
Fiducial generalized confidence intervals
J. Am. Stat. Assoc.
101
2006
254
269
Hannig et al., 2016
J.
Hannig
H.
Iyer
R.C.S.
Lai
T.C.M.
Lee
Generalized fiducial inference: a review and new results
J. Am. Stat. Assoc.
111
2016
1346
1361
Ho and Steinbrecher, 1998
M.W.
Ho
R.
Steinbrecher
Fatal flaws in food safety assessment: critique of the joint FAO/WHO biotechnology and food safety report
Environ. Nutr. Interact.
2
1998
51
84
Hong et al., 2014
B.
Hong
T.L.
Fisher
T.S.
Sult
C.A.
Maxwell
J.A.
Mickelson
H.
Kishino
M.E.H.
Locke
Model-based tolerance intervals derived from cumulative historical composition data: application for substantial equivalence assessment of a genetically modified crop
J. Agric. Food Chem.
62
2014
9916
9926
Kang and Vahl, 2014
Q.
Kang
C.I.
Vahl
Statistical analysis in the safety evaluation of genetically-modified crops: equivalence tests
Crop Sci.
54
2014
2183
2200
Kang and Vahl, 2016
Q.
Kang
C.I.
Vahl
Statistical procedures for testing hypotheses of equivalence in the safety evaluation of a genetically modified crop
J. Agric. Sci.
154
2016
1392
1412
Krishnamoorthy and Mathew, 2002
K.
Krishnamoorthy
T.
Mathew
Assessing occupational exposure via the one-way random effects model with balanced data
J. Agric. Biol. Environ. Stat.
7
2002
440
451
Kok and Kuiper, 2003
E.J.
Kok
H.A.
Kuiper
Comparative safety assessment for biotech crops
Trends Biotechnol.
21
2003
439
444
Kuiper et al., 2001
H.A.
Kuiper
G.A.
Kleter
H.P.J.M.
Noteborn
E.J.
Kok
Assessment of the food safety issues related to genetically modified foods
Plant J.
27
2001
503
528
McNally et al., 2003
R.J.
McNally
H.
Iyer
T.
Mathew
Tests for individual and population bioequivalence based on generalized p-values
Stat. Med.
22
2003
31
53
Midha et al., 1999
K.K.
Midha
M.J.
Rawson
J.W.
Hubbard
Prescribability and switchability of highly variable drugs and drug products
J. Control. Release
62
1999
33
40
Millstone et al., 1999
E.
Millstone
E.
Brunner
S.
Mayer
Beyond 'substantial equivalence'
Nature
401
1999
525
526
OECD, 1993
OECD
Safety Evaluation of Foods Derived by Modern Biotechnology: Concepts and Principles
1993
Organisation for Economic Co-operation and Development (OECD)
Paris
https://www.oecd.org/science/biotrack/41036698.pdf
Schall and Endrenyi, 2010
R.
Schall
L.
Endrenyi
Bioequivalence: tried and tested
Cardiovasc. J. Afr.
21
2010
7
9
Schmidt et al., 2015a
K.
Schmidt
J.
Schmidtke
P.
Schmidt
Statistical Analysis Report on a Chronic Toxicity (1-Year) Study of Rats With Mon810 Maize
2015
Retrieved from
www.cadima.de
Schmidt et al., 2015b
K.
Schmidt
J.
Schmidtke
P.
Schmidt
Statistical Analysis Report on a Repeated Dose 90-Day Oral Toxicity/Longitudinal Study in Rodents With Mon810 Maize
2015
Retrieved from
www.cadima.de
Schmidt et al., 2016
K.
Schmidt
J.
Schmidtke
C.
Kohl
R.
Wilhelm
J.
Schiemann
H.
van der Voet
P.
Steinberg
Enhancing the interpretation of statistical P values in toxicology studies: implementation of linear mixed models (LMMs) and standardized effect sizes (SESs)
Arch. Toxicol.
90
2016
731
751
Schmidt et al., 2017
K.
Schmidt
J.
Schmidtke
P.
Schmidt
C.
Kohl
R.
Wilhelm
J.
Schiemann
H.
van der Voet
P.
Steinberg
Variability of control data and relevance of observed group differences in five oral toxicity studies with genetically modified maize MON810 in rats
Archives Toxicol.
91
4
2017
1977
2006
https://dx.doi.org/10.1007/s00204-016-1857-x
Schuirmann, 1987
D.J.
Schuirmann
A comparison of the two one-sided tests procedure and the power approach for assessing equivalence of average bioavailability
J. Pharmacokin Biopharm.
15
1987
657
680
Searle et al., 1992
S.R.
Searle
G.
Casella
C.E.
McCulloch
Variance Components
1992
Wiley
New York
Vahl and Kang, 2016
C.I.
Vahl
Q.
Kang
Equivalence criteria for the safety evaluation of a genetically modified crop: a statistical perspective
J. Agric. Sci.
154
2016
383
406
van der Voet et al., 2011
H.
van der Voet
J.N.
Perry
B.
Amzal
C.
Paoletti
A statistical assessment of differences and equivalences between genetically modified and reference plant varieties
BMC Biotechnol.
11
2011
15
Walker and Nowacki, 2011
E.
Walker
A.S.
Nowacki
Understanding equivalence and noninferiority testing
J. General Intern. Med.
26
2
2011
192
196
10.1007/s11606-010-1513-8
Weerahandi, 1993
S.
Weerahandi
Generalized confidence intervals
J. Am. Stat. Assoc.
88
1993
899
905
Zeljenková et al., 2014
D.
Zeljenková
K.
Ambrušová
M.
Bartušová
A.
Kebis
J.
Kovrižnych
Z.
Krivošíková
M.
Kuricová
A.
Líšková
E.
Rollerová
V.
Spustová
E.
Szabová
J.
Tulinská
S.
Wimmerová
M.
Levkut
V.
Révajová
Z.
Ševčíková
K.
Schmidt
J.
Schmidtke
J.L.
La Paz
M.
Corujo
M.
Pla
G.A.
Kleter
E.J.
Kok
J.
Sharbati
C.
Hanisch
R.
Einspanier
K.
Adel-Patient
J.-M.
Wal
A.
Spök
A.
Pöting
C.
Kohl
R.
Wilhelm
J.
Schiemann
P.
Steinberg
90-day oral toxicity studies on two genetically modified maize MON810 varieties in Wistar Han RCC rats (EU 7th Framework Programme project GRACE)
Arch. Toxicol.
88
2014
2289
2314
Zeljenková et al., 2016
D.
Zeljenková
R.
Aláčová
J.
Ondrejková
K.
Ambrušová
M.
Bartušová
A.
Kebis
J.
Kovrižnych
E.
Rollerová
E.
Szabová
S.
Wimmerová
M.
Černák
Z.
Krivošíková
M.
Kuricová
A.
Líšková
V.
Spustová
J.
Tulinská
M.
Levkut
V.
Révajová
Z.
Ševčíková
K.
Schmidt
J.
Schmidtke
P.
Schmidt
J.L.
La Paz
M.
Corujo
M.
Pla
G.A.
Kleter
E.J.
Kok
J.
Sharbati
M.
Bohmer
N.
Bohmer
R.
Einspanier
K.
Adel-Patient
A.
Spök
A.
Pöting
C.
Kohl
R.
Wilhelm
J.
Schiemann
P.
Steinberg
One-year oral toxicity study on a genetically modified maize MON810 variety in Wistar Han RCC rats (EU 7th Framework Programme project GRACE)
Arch. Toxicol.
90
2016
2531
2562
10.1007/s00204-016-1798-4
FCT
9310
S0278-6915(17)30563-X
10.1016/j.fct.2017.09.044
The Authors
Equivalence testing using existing reference data: An example with genetically modified and conventional crops in animal feeding studies
Hilko
van der Voet
a
∗
hilko.vandervoet@wur.nl
Paul W.
Goedhart
a
paul.goedhart@wur.nl
Kerstin
Schmidt
b
kerstin.schmidt@biomath.de
a
Wageningen University & Research, Biometris, Droevendaalsesteeg 1, 6708PB Wageningen, The Netherlands
Wageningen University & Research, Biometris
Droevendaalsesteeg 1
Wageningen
6708PB
The Netherlands
b
BioMath GmbH, Friedrich-Barnewitz-Str. 8, 18119 Rostock-Warnemünde, Germany
BioMath GmbH
Friedrich-Barnewitz-Str. 8
Rostock-Warnemünde
18119
Germany
∗
Corresponding author.
Abstract
An equivalence testing method is described to assess the safety of regulated products using relevant data obtained in historical studies with assumedly safe reference products. The method is illustrated using data from a series of animal feeding studies with genetically modified and reference maize varieties. Several criteria for quantifying equivalence are discussed, and study-corrected distribution-wise equivalence is selected as being appropriate for the example case study. An equivalence test is proposed based on a high probability of declaring equivalence in a simplified situation, where there is no between-group variation, where the historical and current studies have the same residual variance, and where the current study is assumed to have a sample size as set by a regulator. The method makes use of generalized fiducial inference methods to integrate uncertainties from both the historical and the current data.
Graphical abstract
Image 1
Highlights
•
An equivalence testing method is proposed to assess the safety of regulated products.
•
We combine data from a current study with test and control, and historical studies with assumedly safe reference products.
•
The method is illustrated with animal feeding studies using genetically modified and reference maize varieties.
•
A high statistical power of the equivalence test is the basis for the equivalence criterion.
•
Generalized fiducial inference is used to integrate uncertainties from the historical and the current data.
Keywords
Food safety
Average equivalence
Distribution-wise equivalence
Linear mixed model
Generalized fiducial inference
Statistical power
KBJ00000000007441
2017-10-21T18:49:29
S300.2
S300
S0278-6915(17)30563-X
10.1016/j.fct.2017.09.044
FCT
0278-6915
9310
FLA
NON-CRC
UNLIMITED
NONE
2017-09-25T15:15:28Z
02786915/v109sP1/S027869151730563X/main.xml
229004
MAIN
JA 5.5.0 ARTICLE
FULL-TEXT
02786915/v109sP1/S027869151730563X/main.assets/fx1.sml
13520
IMAGE-THUMBNAIL
02786915/v109sP1/S027869151730563X/main.assets/gr1.sml
14286
IMAGE-THUMBNAIL
02786915/v109sP1/S027869151730563X/main.assets/gr2.sml
13710
IMAGE-THUMBNAIL
02786915/v109sP1/S027869151730563X/main.assets/gr3.sml
12418
IMAGE-THUMBNAIL
02786915/v109sP1/S027869151730563X/main.assets/gr4.sml
12830
IMAGE-THUMBNAIL
02786915/v109sP1/S027869151730563X/main.assets/gr5.sml
13743
IMAGE-THUMBNAIL
02786915/v109sP1/S027869151730563X/main.assets/fx1.jpg
46700
IMAGE-DOWNSAMPLED
02786915/v109sP1/S027869151730563X/main.assets/gr1.jpg
88436
IMAGE-DOWNSAMPLED
02786915/v109sP1/S027869151730563X/main.assets/gr2.jpg
107597
IMAGE-DOWNSAMPLED
02786915/v109sP1/S027869151730563X/main.assets/gr3.jpg
88819
IMAGE-DOWNSAMPLED
02786915/v109sP1/S027869151730563X/main.assets/gr4.jpg
98075
IMAGE-DOWNSAMPLED
02786915/v109sP1/S027869151730563X/main.assets/gr5.jpg
108484
IMAGE-DOWNSAMPLED
02786915/v109sP1/S027869151730563X/main.assets/mmc4.zip
3494492
APPLICATION
02786915/v109sP1/S027869151730563X/main.assets/mmc1.docx
68436
APPLICATION
02786915/v109sP1/S027869151730563X/main.assets/mmc2.docx
145149
APPLICATION
02786915/v109sP1/S027869151730563X/main.assets/mmc3.docx
2928016
APPLICATION
02786915/v109sP1/S027869151730563X/main.pdf
2075974
MAIN
1.7 6.5
DISTILLED OPTIMIZED BOOKMARKED
02786915/v109sP1/S027869151730563X/main.raw
72026
S0278-6915(17)X0013-6
FCT
0278-6915
109
P1
201711
1
816
S0278-6915(17)30563-X
10.1016/j.fct.2017.09.044
472
485
main.pdf
PDF
1.7
local
1508762898176
collectiebehoudsniveau 1
2017-10-23T12:16:43.308+02:00
local
1508762898688
0
SHA-512
3ceddf21348d87f9e5bf69983cbe05989119885396067798ee9b4d4cecc112950350bb3fbf1f0518b6a2215cde8b975da0d32c071fcf4d44b876677f7b33e0a9
java.security.MessageDigest
2075974
Adobe Acrobat Document
1.7
DIAS
62
DIAS tentative identification
main.pdf
local
1508762898689
0
SHA-512
2a729c72d46b764ed58d664b75d9fe127b196a1b57678a7fa2a9258ac27db0c5267863d8a51eb88d781bb2922df6d759b341e9df709ce1ee3af5b9ff8583cebd
java.security.MessageDigest
72026
not checked
main.raw
local
1508762898690
0
SHA-512
5bef03ebc474fa7118f442375720b84b806518c516520ae3093a4cd8b5f7d291b95a04f59204a02789879ed5cea33c8612b0e55dda7d6d9e27ec46e9176dfdc2
java.security.MessageDigest
229004
not checked
main.xml
local
1508762898691
0
SHA-512
2cf0cb50c03e79e206fc2f622b4b7722f7cbe627f657eac839680c648681012d46db3322d1004e942b353a5cbca8460b63f290c58711dd739b113b6e6fff640c
java.security.MessageDigest
147
not checked
si1.gif
local
1508762898692
0
SHA-512
5c7f8e219741c217a4501587e0d658c934bf65704e04420a3a139e9f6b503f7fb02dad8e95d357b807c26bc43115d5ebb26472e1ad9bfffce87129f09c75d0b5
java.security.MessageDigest
134
not checked
si10.gif
local
1508762898693
0
SHA-512
acbe7b733614c8182be53377d2af696361a77740b34efe1770c9575821ec44ef3e5aacbdca7dea32245820ffee73cc2a22d397b6379defea9e68953ffcd8ba43
java.security.MessageDigest
369
not checked
si100.gif
local
1508762898694
0
SHA-512
48c5a9573c7b35a9f22e58fa5a130dd6de571a96a879bec251a7a58033be81c5d4cf7ed956b38c19ac41b0552f282ad037bad47b53902780050f8edfc08746bf
java.security.MessageDigest
225
not checked
si101.gif
local
1508762898695
0
SHA-512
014f009bb7e339a2414f75693052e111e1b4180cac8f2f78e949fea90ed46471269f786e0cef9b39c8e31c709c3d4c5bbaaa0306d99e5a9bf29a1047c6f59225
java.security.MessageDigest
425
not checked
si102.gif
local
1508762898696
0
SHA-512
dd2728be8a491dbc57e987c3cb6f45eff371a9bbcc63b4b77ec9a52c1a8be7c4f4fee8e0c6fc65cf520d2783b8393ee43721bac5872951e2687b93793e7b70ab
java.security.MessageDigest
339
not checked
si103.gif
local
1508762898697
0
SHA-512
e147b60d499749a4360b497e47448c15ad39e633034e54e4e1945fbd28f5f759b450b6967b0a1c9455817bddfe906e684047ae1468bcd361a07846c4066f7176
java.security.MessageDigest
349
not checked
si104.gif
local
1508762898698
0
SHA-512
786a99d06f0166421414d01a309cf6bf1bc6fd80566180f1da32ba4ab8b343aab71769897ad3f701e6d31e1c302a6c4b6ded0ab118bea029195562a7f8865cc5
java.security.MessageDigest
336
not checked
si105.gif
local
1508762898699
0
SHA-512
39501c465d59d3a920a4adc7adae67363cad545d76dca7b3ee87cbb2cf0077eadf6c9376353b634bd2db272665c89ac1b8676da4c0eda460c794f1321ea9f0e3
java.security.MessageDigest
407
not checked
si106.gif
local
1508762898700
0
SHA-512
624bcd72dc7762117eb43e5ffb04d2d95e87f644a9e95760cc076c3b5c581dec373d383002ca8bee45bb9a81c1f8a11d3a11aa3c71a6f69321ce586f84a48fa3
java.security.MessageDigest
838
not checked
si107.gif
local
1508762898701
0
SHA-512
5167c9b79a7df86013b977b4f8084a17d1d8f2bf48ea60694a6821732c10dd458896fcba22b67291a9006ca07de18d86abab45625046ced0851781e814ad4587
java.security.MessageDigest
593
not checked
si108.gif
local
1508762898702
0
SHA-512
b9d24f322ed2853efd2a48349cf1b3b8fef6c3065dcc6bdd7c3d68b157e50a6e44fbe4c78489fe5cbb88fcc0900a3dd65bb5ecf488b461194070c373f2f80527
java.security.MessageDigest
1269
not checked
si109.gif
local
1508762898703
0
SHA-512
c7a061f196f0efc3477d7b75f7e8f119a5493e7d7c96a6ee51f2d78053b3fdfec6cf9f5ff8ae30b68ba12e6cca087e952c408c08acf89521e2055f0718b55b7e
java.security.MessageDigest
250
not checked
si11.gif
local
1508762898704
0
SHA-512
2752408aa90dd993e9625213d6e6d7c76d1c9c15dc59c5c6f8988e37f42c11c800cfff201cd06909f7a5a00305174dfe34a07ddbf8372f1f6fc40b601f157287
java.security.MessageDigest
325
not checked
si110.gif
local
1508762898705
0
SHA-512
f98fc08cdca9ba83e7698609d5c5070214c58bcdecad8d7c5b658ab10e8f56027cb0f593ebe177d90fc5e326d3ba183b1f707d1ce1d2f79f3b37db4d029b66b6
java.security.MessageDigest
141
not checked
si111.gif
local
1508762898706
0
SHA-512
d5031efa8b0a7ec7db6461c96e7f203301ec86e9399dfc80ecc0a6664cc07d7f28c8cefdd9831fb1d0cf475ff2903f27bab7627de108be0c53068e12852b119d
java.security.MessageDigest
1565
not checked
si112.gif
local
1508762898707
0
SHA-512
e773c9b67a3e6e4c4e55d5547849d0662cf50e2f9e10c8a59e0c71079d76f3eb57f88098871ea4da962335b84fdbe7fe5f00ea4210405688a5c1e6b579173c64
java.security.MessageDigest
425
not checked
si113.gif
local
1508762898708
0
SHA-512
c4f0e3d795d83cbc6c156d123961f6f697c7ef3d2d67f24269ad4a1beb521ed28497077e7bbe7cfc19324d0188b27170e83149a3a2248911918582fb4297b08e
java.security.MessageDigest
314
not checked
si114.gif
local
1508762898709
0
SHA-512
f4d8b5015169c37926fe7a1e39bb676d2436caee8cbd2d3b4e6b7bbf2a76941575eb68050d0c8b70777d503687afdbc79225adb301fca01f0a4287b00a7d4984
java.security.MessageDigest
702
not checked
si115.gif
local
1508762898710
0
SHA-512
8f727e6c2dd2473f9e74bcbbfc34edbc03412d7a28e6be4f7e2d6db6eeef9c3bd4af54e480f9b3e4235b9a083766199b990bf17a8c293a18b23191b018eff5d8
java.security.MessageDigest
147
not checked
si116.gif
local
1508762898711
0
SHA-512
6fe34bc775b0a7811890485817869acff4485300e6a419e5e0a099a04663469c84354cd182288d6adc8d7e9dc6e4bcdf09a80f8eb55d1a939875a804053b48dd
java.security.MessageDigest
207
not checked
si117.gif
local
1508762898712
0
SHA-512
19268c18b875ddbdd163b99ba660d453126d8e3febb363a30702b07d9bd89fd6b47b59eeabf0cb658a1561445ad77d40da5e5dc831a7243d53c6986fdc5a330b
java.security.MessageDigest
234
not checked
si118.gif
local
1508762898713
0
SHA-512
af7d785a92c6023e050a36f552fa249c3b1fc1bae55d57be7aab83502ab69781dc6470700c98e7287da4555c0490f76260cc694a6da28e27e780028a0255c3a8
java.security.MessageDigest
251
not checked
si12.gif
local
1508762898714
0
SHA-512
a534b688740180cbceff0d2513769f3039d79e23d6d17f91759de1ff152160d4879ef239fec0a1c68a7b55586d5afc6fde24c162856725e6a01b2e44eb877812
java.security.MessageDigest
187
not checked
si120.gif
local
1508762898715
0
SHA-512
0b7a85f27189ce2d211ca788f0e1cfe511673fad824bdae40ece96dcf88f94608615d2afb01b8ee28b4ef7e77ea440c52c25e41846f2ff1b9457809ba1820e7d
java.security.MessageDigest
330
not checked
si121.gif
local
1508762898716
0
SHA-512
d5bfac622641d5eb9d6c0bc7a4fa59eeaecb5e00595eea4f2d260a490ae68887ac9be63943066379161ad213f8ccb421f9c7964ec0d77f2f5de8a36ce540acb3
java.security.MessageDigest
247
not checked
si122.gif
local
1508762898717
0
SHA-512
232e52882f182f89a494742d5ef37cb8da0de858a993e99bf3bbc336d1420cd8de5d126149ea34c013b65fae4538fa96fa8325b003fba4b3efacdf7b508e01ac
java.security.MessageDigest
283
not checked
si123.gif
local
1508762898718
0
SHA-512
13c381f204023942dff5e8920c74030d6e428717c1c8e0d73723490692536b8305605d929079cd4867d17751e0b843ba495439063bde61d191e8d1c7f01db1e2
java.security.MessageDigest
154
not checked
si124.gif
local
1508762898719
0
SHA-512
ec0db9c3e10fc9422240ce1b79835cd60e5a84fb157d2bf598d4509332f137b814c4a4f3ee4dc1cb09c8bd62c1612e45d7ebc3dfd2a448cb98b4f4badea6b0f6
java.security.MessageDigest
353
not checked
si125.gif
local
1508762898720
0
SHA-512
376f8b8c9c76e0aa16ea84e3c643224a8bcd22c1743b195338cc368833d5c5c7620268e86328572b3f0e348a540161582d34d94afcf38816126c636ac9da6f10
java.security.MessageDigest
350
not checked
si126.gif
local
1508762898721
0
SHA-512
423932ad16e266fb330c354e97a236d3c8fccbb64a1bdd18814fd6f64e8dcce6b841136f10b7a4fe68ac7363a6851c1799d4fe954a88cdea2a643abf77c6cdb0
java.security.MessageDigest
527
not checked
si127.gif
local
1508762898722
0
SHA-512
16cd71853eca5c910b1e65d6b670d469140ecc7857b5284b3c6c08992103ec6c4f01772cf5cbd4d390e2cf87976178695da4c87648761f641bd5c5fced39d9e3
java.security.MessageDigest
468
not checked
si128.gif
local
1508762898723
0
SHA-512
5b7041bf5fe06584003ba1a380bad55573c0cf67906b20923989116b45d6aaa68a465b22a02932618d3ae68e4759ca14a115b2ec7bbfade2f9d3315107d3d0fc
java.security.MessageDigest
473
not checked
si129.gif
local
1508762898724
0
SHA-512
37cf6dbf4eff2cc91d154db7e0f77c244c133df519bcfd5932962a3031137bd45a70526f98fdfc19327853025385f8f0a5a87c9bab5ddacbe75f0ba6fffff044
java.security.MessageDigest
178
not checked
si13.gif
local
1508762898725
0
SHA-512
07d77f000495a5d4c1076259c33fc273dac0ea5b354a138586762e454503d1f1527a103bd20c33d813fcf518096836c820eb848a9a670c76aa471379fda25ed6
java.security.MessageDigest
535
not checked
si130.gif
local
1508762898726
0
SHA-512
466e2edf5f45307bafc5d65e3020258db75609affc993536a6d4703b66905a4e67aa224863e0cfeae43f0f22cc7c80ecfa47b96b17156e085e2f7ab065caca7d
java.security.MessageDigest
235
not checked
si131.gif
local
1508762898727
0
SHA-512
3cf380df9afc1525ab72f5a6b5cb245085ee3b8befc3cc95b3e797bdb653e64f3e6ef4085f68b9e4663cd055ce7939c5610559f7eee7213c358061a24bd5f28e
java.security.MessageDigest
454
not checked
si132.gif
local
1508762898728
0
SHA-512
3ab515772081c2cb5ba7e21299f7b6474e86b140765dbf58c4008479bc3f88afab14c2f039346c3add60b91ec10549205734d99bf43819e00618accd159f1811
java.security.MessageDigest
233
not checked
si133.gif
local
1508762898729
0
SHA-512
818980fa97db1b28912aff030d3e3901dc7856ae0c82216c29707793a47effc9542973338b0b82b8cdff7a79c9f1e7244c7b43065243dcffd9fe208bd6daa50c
java.security.MessageDigest
259
not checked
si134.gif
local
1508762898730
0
SHA-512
62a32bb6635d10c4c0b6562580a62016ef0cab6046aa4153fc2d4998be509920e04270f7c48ede487d2d76d8339621e8a7a2cb7f2e49d7abe739dc55e8787a25
java.security.MessageDigest
172
not checked
si135.gif
local
1508762898731
0
SHA-512
34f2f1c0b13966ee82283eaa532dca3c6e3b470acb6ab36ce5eb768c91d7507319ae5473e32d3cbc7dd13575a65d6c855f11da40f0f8816c77e73b79d0ba9bc6
java.security.MessageDigest
697
not checked
si136.gif
local
1508762898732
0
SHA-512
fb69d9ddc4144a2ab436e23b878606aed60d0bfa20c84cf358d8eb6654fdccfb039b8283ac20198794cd86930f9c53f6beb8d04b9c82a213b72532ca887ff544
java.security.MessageDigest
252
not checked
si137.gif
local
1508762898733
0
SHA-512
656c4552ed58d03567b43d86b8281700cd078c2c9f169db2c83f40c55c4fe726e853deaf439d1deb43fcc66f7c6126483c3f476e74a082a90c997014a7e3b050
java.security.MessageDigest
196
not checked
si138.gif
local
1508762898734
0
SHA-512
90735543789bb80a2db3ef1dcfd43b316ba583a2323fcf6cedf03215d3e1282f48798f1aaf94b43fd3cf107e7fae5a830e876026cffdf318bc47fd931afaad6e
java.security.MessageDigest
243
not checked
si139.gif
local
1508762898735
0
SHA-512
65c3abe98d2e5305b36f9a2e9181e13abc3566c7408544c49cedc19d8630ace7f90a9bdcd7d55364180c686d411cd21d7dfb3f5ba56b6a1c0f39fcfab41eb8dc
java.security.MessageDigest
182
not checked
si14.gif
local
1508762898736
0
SHA-512
1e578fd05b43f655af69193d3899951eda5724e66922578d5de5f7dcd718d6ee18688e14d4ea185723af9e1e85f5353693234dd0a87722db8ec6198d377c9166
java.security.MessageDigest
173
not checked
si140.gif
local
1508762898737
0
SHA-512
f999d8859efe84e482a6cbec8986a14462c9cce302f099026f3b0c2e315e05bfce634564b1e24ba65d0ab6769716eb55e29ba0decf7db17cfa2091167e5cc4ba
java.security.MessageDigest
584
not checked
si142.gif
local
1508762898738
0
SHA-512
632c79c74302fffec987f58bfec4c8b2717847fd16a47edc1320a81a3a0af845051b6e0bf1b8c870b0a1fff0163d11e57bb9df1cbc79a8ecdcbed56b4040ee21
java.security.MessageDigest
208
not checked
si143.gif
local
1508762898739
0
SHA-512
6fa9e8c3ab44ff94ab49d1280bdb3e927ed738681450d4330c7323fbe8a0620de717349ed37230ce483251a4947585caff2b18c6c20de0bdfe0e2a5db5c24527
java.security.MessageDigest
304
not checked
si144.gif
local
1508762898740
0
SHA-512
749edc467809a31cbfd1c11a7202c756a574f50b4b9d7a21be748bb87ee4ce7a84a4ff274afaed0951cb6a022a33365669fd371039780b1df7c2d9222a2b4bb1
java.security.MessageDigest
597
not checked
si145.gif
local
1508762898741
0
SHA-512
0720b68a25dc7d5426c8bc8e4cc90f8311da80f126107474a2331f3df03916934b01c43e36eab9f806b0bcc3bfb8aaf7385a48251227a29eede10dee2d0fbcf1
java.security.MessageDigest
1025
not checked
si146.gif
local
1508762898742
0
SHA-512
a12acccdcc44c13277a4ab5a9f358efc7cec278259655584e0962374bbf5374aa9afa56bdb99ef12c169d3ace7ed1141641a6f32c542d913300b17f9b79631a8
java.security.MessageDigest
2028
not checked
si147.gif
local
1508762898743
0
SHA-512
a85153030bde35cd202a9b6865f78aa884c5c991c1f762f613dac27c10d1c8e5ee1f946c6007d60bd6415b602c58e49cac6199ea136b58e9b81ce80217de9379
java.security.MessageDigest
383
not checked
si148.gif
local
1508762898744
0
SHA-512
c3d9e8055f7f278742d3bc581037fd217e0a87a4e0f94a6b2141756ef69b2c552492cd9c5f9cf3733a5af5da2feed6e825856d29c61ac44c6c51da01f86399db
java.security.MessageDigest
1463
not checked
si149.gif
local
1508762898745
0
SHA-512
a8806f62a601ca1ff283a42e9b228224634cf68be31dffb37b2c19b94fdeb8bb3e7e07d3dff4e97a66558ebaa1cfd3445f61e47caf58ff91b220f6dfc86734df
java.security.MessageDigest
1918
not checked
si15.gif
local
1508762898746
0
SHA-512
7a61b0c186fd4f0c3eb2d03a57c483623a61792f6a373a7b9f75a425b2d2a0348bb44af4359377b5adae635154b486aec37c9241e5578e8a6ef4ae145dea2189
java.security.MessageDigest
324
not checked
si150.gif
local
1508762898747
0
SHA-512
18da544f087f3e06d7535964d9075a1147540fe334e8f8ac8827ed240f645b9fbde4b172582e595ceae979902bc2efb0f99f0b998c6200879c580b1a4b1dc248
java.security.MessageDigest
133
not checked
si151.gif
local
1508762898748
0
SHA-512
139c7511c1fe88cc0f25750b55f46750296d3f0c7371558f81aca8c9694c6bfbde32700ec6c5705730a9a160a0dfbc89fe6c09d14346ae4ab40186b739d3a3bd
java.security.MessageDigest
143
not checked
si152.gif
local
1508762898749
0
SHA-512
3d910a31ddd4a466a061d1b9bca00eed11d9515064672a90fa6d9346fffa0bcb901198d97415a7e805c547a616333e2fa382e15b94e87e57b6227016091f1a85
java.security.MessageDigest
293
not checked
si153.gif
local
1508762898750
0
SHA-512
6234bf52d387a0189913034a4c6d8d272fdf2adab847d93fb8399b837ce101f23b9b0626c021f18c94d5c8826fdeadcf6c14ab37ed399e8580d32ebc9ae4d613
java.security.MessageDigest
373
not checked
si154.gif
local
1508762898751
0
SHA-512
26cfae71d8df788afcb35100e79e77fb9810f50e3fa9c91c0a74338b26a5204fdf89bbee4f93666be89a0ef46a20246b9ca06b3f92f3398242425c0c477ccfa4
java.security.MessageDigest
231
not checked
si155.gif
local
1508762898752
0
SHA-512
0e65ebab40279c82e024fe5e4c162c1f28cb6571f1154f91df11d9857d9092daee0d87063d6e39aa5aefd2b49ad18a8698ab52664edf8eda6512a653562ac33d
java.security.MessageDigest
251
not checked
si156.gif
local
1508762898753
0
SHA-512
11aceab08463cfe8e4662422431560684f772549688d3b689a40f3e1947e2a8962ed392b7b93934df76f1b2619e204ba016d21ec8720c19ea9222ac54325dff0
java.security.MessageDigest
1038
not checked
si157.gif
local
1508762898754
0
SHA-512
a7a3f3258fb90194c0718097fc0d7cbd7c4bf185f2167bb2758dc404cdff7d216a2f4c0f71ae0f014db9ab3cc1553754bf555c85daf5c17cfc8dad2b69cfb49d
java.security.MessageDigest
431
not checked
si158.gif
local
1508762898755
0
SHA-512
72aaf8fe0b37aa976d776657770369d6ac2633918ffab6b4148d50b278c8cceb61d27427df14e562cbc9f3abe482c07468d6fc6a58c7fd481922555e131d921e
java.security.MessageDigest
328
not checked
si159.gif
local
1508762898756
0
SHA-512
146ffde7cde0400cf1103885b2f437f53a7353fc7f90a8ce2d4671b3af1fcb8bb4928b0439dd38a8e46a8fdc8dadfc47671a03e609bc67cc070592caf014415f
java.security.MessageDigest
175
not checked
si16.gif
local
1508762898757
0
SHA-512
81cf309d3d7741c9f4e38b5a794868df5f8e8f1980313d1225c31e178d4011b9b2d1413e34cfede93afd295eda461f21923793eff19faf3c58fee20896feae44
java.security.MessageDigest
262
not checked
si161.gif
local
1508762898758
0
SHA-512
129739f7df5998891f4b53ec2f2b4697e32f7b7a970d0db2d7d3fdde042786c2465880badf8848faaf11533eb94fd1776c108adc2a728d90c5bf4f2d3112c2a2
java.security.MessageDigest
3606
not checked
si162.gif
local
1508762898759
0
SHA-512
135b555ee55853f1e7dee4241eac61b89adba16281eb5328f67cc1bb7a583aa2c8b9fb031e587d173c048cc8c6150f89bdd896f63eb1328bdebafaaea24fb250
java.security.MessageDigest
328
not checked
si163.gif
local
1508762898760
0
SHA-512
4ba4e6dd9747d6ba3c3bef4dd0dda0a08069e35b111bd13552928ca99189ebb5ceecb3f541b18ece787f882363aa025a155709137b37ba9890f4ce69b30574a6
java.security.MessageDigest
302
not checked
si164.gif
local
1508762898761
0
SHA-512
0c41bfd18a8f4a168c6ccf40476cb9cb43dcff9eb2e8e337474cef3fca4f28e929214e818c4186837ca799ca8f3b06f4998bc1ce0736352bc6ff65d6926a5886
java.security.MessageDigest
1354
not checked
si165.gif
local
1508762898762
0
SHA-512
342048a4ffef449f77a5ca99dfc2a1db774f3a696c05ce702d0715cc8d8ba82266d2b129b7ef1a8c9e6f26cd2e170e66d5a8229c9731733d9691beb79571a245
java.security.MessageDigest
389
not checked
si166.gif
local
1508762898763
0
SHA-512
ce018b191f5b9415e16a44cb77dd8e928ea1be2399b12e600fb4bb9e675bce4736b7492e6778c40ed995b63ec3413fd2ef9f6b26b73ee208bb009add426796f2
java.security.MessageDigest
448
not checked
si167.gif
local
1508762898764
0
SHA-512
4dfef5743411df9f5837bc9bc6aa5c0fbe5313839d06ee140f76823a832a278ddeae538617a2635abd1e2061ba17d0a434c41b091eb0816a58235ba0e8af53b5
java.security.MessageDigest
585
not checked
si168.gif
local
1508762898765
0
SHA-512
f88d80b03eb8d51300ad13794f9b644be02e07801a4385ef969c86035fcab8eddbd073b04ff4e6ff657c4cb451f13c9042b129465a26a5ac6586faa9e7652273
java.security.MessageDigest
2883
not checked
si169.gif
local
1508762898766
0
SHA-512
e8639d8fbccf0bf06c8cd442b3da21d1a2812a253117ea2de411287b0c2d919c483e3d2824b5787fcb6e8469da2be59241ead5418c1a34607ecf264db1056ebb
java.security.MessageDigest
163
not checked
si17.gif
local
1508762898767
0
SHA-512
d05f03faf8368935f7b02f12536af0f1d40acaf7b526833770feacc57f517c6433f901f7a4f252c7b4c26b9c65ea82001ce1e5315aca362e035dd144319ff876
java.security.MessageDigest
405
not checked
si170.gif
local
1508762898768
0
SHA-512
82dc13a26ee8686daa4f64415c3a067d7e9c06b89410b6ba15bcd0ffdc9533ed4916ad737e5061e3e7d59d645aad32ad2044a4c0fd2a4c71507fe1528c77fcc4
java.security.MessageDigest
264
not checked
si171.gif
local
1508762898769
0
SHA-512
384b7af02c279d4ade2191c2bdfc66038913dae0df98ce14634753ca6b0dd7f6a6067d66e586d64c1b32fe4c9ad16d1a45369faf02fb72090170665bdc8df02b
java.security.MessageDigest
1810
not checked
si172.gif
local
1508762898770
0
SHA-512
6bb17a0243f1ef9386a00da39655f78ba33aeb396cff06a6fb64550ccd50ff6acde8ad0a47bb1bdb62bd69fb82f7e8299a1586501219c08f8b6b42eaaf4824f5
java.security.MessageDigest
518
not checked
si173.gif
local
1508762898771
0
SHA-512
088c38ec131f8e0f594123ca80a69b4ee34f0412497189277ef25ce613dd3078ff6e31d8818bd9a0e55af3b29e9dcfc99d79b99c08c5c9db30aed9de5a527604
java.security.MessageDigest
225
not checked
si174.gif
local
1508762898772
0
SHA-512
d1150d64c7f7d5436b9b0ca28845f6c8811ab6409cf20246281dbb1ffd88128e71f66b11e5a5e8ad3c81edfff075613fa4191730f4583a5bad0b0636956df4bc
java.security.MessageDigest
290
not checked
si175.gif
local
1508762898773
0
SHA-512
8284754ba7dedd8c71f48afd34392373137fa941d836c72a800a1120ab28dfac080cf42e02ad58309a2c7075f92c8821e31589b172a67635ca34997f52a03cf0
java.security.MessageDigest
276
not checked
si176.gif
local
1508762898774
0
SHA-512
bb3ebf10a00abf86d341c0892db969751e758b5f3b7949642b6552c3042ce1997cda80027212f7485fb0f464dcad5589cc01d93bd910fc9c892c9ab653ff243b
java.security.MessageDigest
207
not checked
si177.gif
local
1508762898775
0
SHA-512
db0fc98035072a5605b15e5628e1f39114b33c918b6a132121087fef867e67635d4de8df9fcb28faf85f852bca8dfcb936b17c3878fe03ee5e4cdfc54d00bf4a
java.security.MessageDigest
155
not checked
si178.gif
local
1508762898776
0
SHA-512
bf674a56077beea130fb76f3f3b245517f6cc7646bdd71d77e798cc2ab4a77fc5de794ebe3e16b4297c67ad31d825e88727137e4ab1c0468b358148bdaf53a4d
java.security.MessageDigest
156
not checked
si179.gif
local
1508762898777
0
SHA-512
9a9503d5f9b80956f4f6418b67f6453ef6443ce4a40432b11e12f6787994ba8d6f64db061939a66c4e0f5c17906d0669aaf8bbfdb4e2db7dbc1d4189feb24bdb
java.security.MessageDigest
172
not checked
si18.gif
local
1508762898778
0
SHA-512
075b74a2833e8659948b2c18633cf5dc72e20a5c8a645b9d5dc02360f4b303d43dd4933f17615dad44df52df49f7bec296ee06847d646d58c85bf1d51fdd94c6
java.security.MessageDigest
196
not checked
si180.gif
local
1508762898779
0
SHA-512
94c4cfca71c5adc325cb2fa35af375a61e1d33dc589c8de2fccd70e072de86878891870494dc803401ace8bf44b2d9b99215279023689bd4082e475eb22beab7
java.security.MessageDigest
243
not checked
si181.gif
local
1508762898780
0
SHA-512
9055380c4cedad9e9daa79bf06324d8e0d1653062adf7c575b8aba0f6fd50f159d55348b2da5ff856e3ea92d0e45f1964b2b3dd1e52ad36470a4a39329701b01
java.security.MessageDigest
238
not checked
si182.gif
local
1508762898781
0
SHA-512
0a41a99d401e06d9fdab7855245d25b54d24a669319728a699a441823ac611c9a581158f344bf4a688082e01aecf404a23acdd3ea334c1d30f89fb4fa1133ab5
java.security.MessageDigest
310
not checked
si183.gif
local
1508762898782
0
SHA-512
5d215c791558e6a70cce165902f1a9fa12c3339c19f474fbdba707e38d2052d7b0e6d3d3f4e24721b72c05929143d081648036d8bb6715e309e2051d2d8b3279
java.security.MessageDigest
398
not checked
si184.gif
local
1508762898783
0
SHA-512
be9fd9459630c9b4094170b13f4c2f573987f3cadc295a4b12ce0f32edbb691e4e05f709e60e3cf5a1df17a7d5caec15884f9a7cce199fddfdb578e32d2825a7
java.security.MessageDigest
446
not checked
si185.gif
local
1508762898784
0
SHA-512
7ff9e150045f74022e6ffa5de1c25df8790c7b5ca39dee4f5faa53ba392d22c427ad794cb354f40b3fe1fc744991e1ca68f093bc8187d19a3088fa8da0387d6d
java.security.MessageDigest
296
not checked
si186.gif
local
1508762898785
0
SHA-512
a94877cb7aa0f0eb63df3cc0ab4c7ef252eef4bcda92108cda3b45276bbec65a8ab2343af7164e230d62439442b899a1e43fa3c73aa61e283e0649813daa8974
java.security.MessageDigest
599
not checked
si187.gif
local
1508762898786
0
SHA-512
a65f4d6888d0bd95da5ad92dd37220a53e126bad84154b71689a6c8ff592e752e59be372c5fe420a41898d44fe154a972e925b7e260e7da3d0983935b2d0bb0b
java.security.MessageDigest
328
not checked
si188.gif
local
1508762898787
0
SHA-512
8462a9c6003a9123522702e14aa02a6bddf46ce8f655d65b0e8fe7cff95c3081e7a2e29ab496d78ad207c60cefe59bef20720cebcd1edb4c8c91f76c7741f3b6
java.security.MessageDigest
458
not checked
si189.gif
local
1508762898788
0
SHA-512
435b6de77a907bd016ed53da53c22d8ba3e32c01fa856a5f578dd6018adf836412980b5b8ae2f6ebefcdb6c1030991f1e6b99e31e3b030fc693f931a68c01403
java.security.MessageDigest
132
not checked
si19.gif
local
1508762898789
0
SHA-512
460f306bff17d0d912543346e4b844fa81aac1a83f25b8545b39544f893128450c20946d74efe48b6f66778238ef686f38760de3073fa270d7d0d18ca9e2bd03
java.security.MessageDigest
387
not checked
si190.gif
local
1508762898790
0
SHA-512
5ecfa408bf486960f54e509c04f01ad83fc619ca9cbafeac84610e31472523465df0a1c41604e93dd0a5987118b913adc11519767168bfb5f1488672bde30f86
java.security.MessageDigest
152
not checked
si191.gif
local
1508762898791
0
SHA-512
38c48bc4c0ccc1263082a2145ea1c0ccea37420f2e3c449ed931fc3d2f054365236ec46821cc018d45a77a62a0f1c00fe97b4156c1f2dbe25473cd6cd6b41eed
java.security.MessageDigest
149
not checked
si192.gif
local
1508762898792
0
SHA-512
631568392f4db1524665ed20cb910420013d5091d957b2241507fb46675b91614a8de9d21fdebc2d679dc0c40c6a48c29f518215b78b70fc1820551e5839df3b
java.security.MessageDigest
377
not checked
si193.gif
local
1508762898793
0
SHA-512
041b5385d6daac34a1653f952c748c257f38b4e4732bec27768c086cca76454b7683f9bc094c37867f8f4f92b5c032e460515882207545eb976b62b36192065f
java.security.MessageDigest
146
not checked
si194.gif
local
1508762898794
0
SHA-512
ae719bd4f5e2ed57ed40ce6b4d8d6d24672d48042b85713f0d37ed458cf1cf057d94a59f1db7ee7d180e91915ac38332ab340c186bbf7e735f8b973d061d991a
java.security.MessageDigest
152
not checked
si195.gif
local
1508762898795
0
SHA-512
c27d90273ad9c31ee49cf8ef9e2869b133511f4eb57b434cf922126ca13d12bb7323cca94ff60d5ba889dfdcd0df165269fd46cc83c333b363ccf09036674fd5
java.security.MessageDigest
202
not checked
si196.gif
local
1508762898796
0
SHA-512
50bcb18fc630e0ece002859899d0aabc9ae8c1563e3b09b6bd26f378e62c7daddba09b00fc226207e8b3a3a82fb6b6e23baf57c1f08f2787cbb115994807a9b9
java.security.MessageDigest
150
not checked
si197.gif
local
1508762898797
0
SHA-512
9bf51ef967f251dd3326a13de0ba88610d7c36a67144a5cae218e436254ac5d24c27acf2b627e8c43d2ee9634e2735246acab7462a8a12da78e2ab81862ed61f
java.security.MessageDigest
153
not checked
si198.gif
local
1508762898798
0
SHA-512
acbe9fa3458bf5fcccc42dd4ffdb1cf40b493568000677fda179f4f210e8cf728489e0db696d56285431013a786b2889b5a6d98061520a9a7125f27df20e305d
java.security.MessageDigest
207
not checked
si199.gif
local
1508762898799
0
SHA-512
4d39e8f5b0b0bdc981aa7cd502cea0942461f70451da1508becb3df3432d96b183943bd4ce0d450513339c84fe8db94b2f8d2863592994d907dcc5e8fee4a12a
java.security.MessageDigest
135
not checked
si2.gif
local
1508762898800
0
SHA-512
0843c27d5de9b60540fa8ebe79e99f784aa3238f095f66abbfb5056e91201249a7f342be7a62363488454e7bc647b90f1c5c247bded903226bd30b1c92b601f0
java.security.MessageDigest
135
not checked
si20.gif
local
1508762898801
0
SHA-512
57e25e5057dd17736d2c16d6eee0dab910a839f3f82ba7f891632eb7322808d2a174c932dffe0584496efa0e929babb0983c926d659027ce008f79873e81a603
java.security.MessageDigest
205
not checked
si200.gif
local
1508762898802
0
SHA-512
5977dcb67ff0fdd9ae024747d01b27c4c8fa6c96f279bcb783fe68a3fbf06599587a1b165c7b68f1c5d1bb5a0efbf7acbd1590c02832ef8e90c9b0c1be5943a8
java.security.MessageDigest
194
not checked
si201.gif
local
1508762898803
0
SHA-512
bc826a85678a178407c94b361495bf545c349d6fe4871df9a8ce0517ecff19f22ea0e1a5d1f17032c60c3208dab67f0377d3997afd56bc8d5123b9c3ca101c86
java.security.MessageDigest
377
not checked
si202.gif
local
1508762898804
0
SHA-512
f48142cec31a6d620d6c1014d901647078e54e47aa783542dfaa4b09e4bd80b21860457e2726323adce060044beeb09fb42d98453050f4f7335638b08f0257f4
java.security.MessageDigest
215
not checked
si203.gif
local
1508762898805
0
SHA-512
f7ef116730999c06986fefe437940d25b0fd5fea87226091589f19392764916f04e3497e58311a234f3d5d7e24e2a57f758321b250d917ac7955882ad1fccce6
java.security.MessageDigest
287
not checked
si204.gif
local
1508762898806
0
SHA-512
d896a6ef4a4d752cec81ea8887e5ed2dcb131cf6223bc430df7e4753619c325cb9fd2eb13bebd5f400bf28da561696d5633a2ef0662bb19fada8905a8aea0c5e
java.security.MessageDigest
229
not checked
si205.gif
local
1508762898807
0
SHA-512
9d024889f7d22bc815b60260bc7dd284dde94a8fff3e2aae1fc9a462bbac4c9e43ccdc591c3f8f6a3677611c2e2a8181fcb914e9b72299dd131c41559faa8458
java.security.MessageDigest
242
not checked
si206.gif
local
1508762898808
0
SHA-512
b2ea2d50c5a897f6cf13ca7939a765882874d603fc23f7bad215b22e4ac4ee48823fd210783e15a34cd75588b49de9ae8ebd502b72b75c1dd4be46379052b424
java.security.MessageDigest
245
not checked
si207.gif
local
1508762898809
0
SHA-512
2d577f94308c5e1182e36332534a461db51b9d639f89b6367d6cf7609db9980abb73018cb45d2b0e511ce8f8c7ec8443a72fdd96c0541848b89ae5bb0ae2573b
java.security.MessageDigest
227
not checked
si208.gif
local
1508762898810
0
SHA-512
219258a4df63e30fd489514589cc2b09d9dc489723c0662929086947b5bf3c24ccdff01de9fc9bc595252efcb3d21a02c9cb080b597be237dfc17b82167c8e15
java.security.MessageDigest
151
not checked
si209.gif
local
1508762898811
0
SHA-512
6150f9472102ef96b7ba356ab9ffad205a5af0e336302f0474fd2250a0722e804da10f59fe37d5c28737bb6ba7468a3f0b0767aac0f18b83eea0505ccee83ef8
java.security.MessageDigest
155
not checked
si21.gif
local
1508762898812
0
SHA-512
67e3d807cef5d7853e733a336490b9b6db5eb17d9a720dff87446b3cf29a504415eb364e529c09ede6132b3c2981817aa460de6ebdf787bc92e81d32dbf3301a
java.security.MessageDigest
210
not checked
si210.gif
local
1508762898813
0
SHA-512
226d5475f251859c7497349a5150a951ea5f1becfb53a853a83aceb4ec2042172143af1557c68fa9c606383a77bf82fe14b0b09025a8d3894ff0ac7e9bec9032
java.security.MessageDigest
217
not checked
si211.gif
local
1508762898814
0
SHA-512
dd287bda502d52311230c0728f5f7710cdca5ba499a75b28d055285ef1865faba960249af90bbc11ee75d5b101179e69f5b15cb7c87bac2c6e6e2088e10f6286
java.security.MessageDigest
216
not checked
si212.gif
local
1508762898815
0
SHA-512
837fe38faf35484c5914755eadb4627211fe908e0f676950851666fd460197a62272dd357f6251963032e05a9f67ac2a56470a24ee1a4db7ab5c033c9e6da057
java.security.MessageDigest
210
not checked
si213.gif
local
1508762898816
0
SHA-512
13c11d1dec31fffa42f10e93e78b6180088f52c15a6ad90564b51cf0839a44dc5508478c0477edda5bb3aedf0a0e117f3cadbc80af2c449c97d9f8e84f8a9cd1
java.security.MessageDigest
211
not checked
si214.gif
local
1508762898817
0
SHA-512
65db354ff3df2de5f4f056021d5e57fb8a02201a30a85338f19b860a28b9d2fe162aa5774c9d65d18a4355ecf843b950a922cf0f0246a6cade53522201821de8
java.security.MessageDigest
486
not checked
si215.gif
local
1508762898818
0
SHA-512
feb5ac6908750fb197d6ef38dc1f2958ba34f98a921ae229b4766e5c99b229af84c60acb076e97577aced1044a28ab93e85b813014d64746464a55931570a1f8
java.security.MessageDigest
158
not checked
si216.gif
local
1508762898819
0
SHA-512
b96fe2913aa9355c94d8465a81fcc6a32cdaf180acb5f9c0a0fd9b92e678c5c0e6e5c74bad6d5f6e9e089cb2c116475f61278a67031487552d56daec4220ee78
java.security.MessageDigest
728
not checked
si217.gif
local
1508762898820
0
SHA-512
c1dfb282dbb478bbe939509d3aebd88ebe978891eacf7d3688033d1fadf0808e192fe03a6ddf9f0d251f1908026e69906aa25444413c30c0641d4058372bc548
java.security.MessageDigest
373
not checked
si218.gif
local
1508762898821
0
SHA-512
8f65e82d1ecbbbde48b7d0984e30128b0baf531f188868a243d7b5f7efedf9804c660adaf531314798c883f66726caceb1863ab15dbcb190479521a0b0d8223d
java.security.MessageDigest
578
not checked
si219.gif
local
1508762898822
0
SHA-512
1882e88aeb84c32a831cdc5dcf85c5f9fba0470a9ea6e337689262cd7b944995786bbef2eefa923b6205145db6da26eaa98398ea4cf39dadab83f5825848a868
java.security.MessageDigest
183
not checked
si22.gif
local
1508762898823
0
SHA-512
cc8283158747422694e8ae66a9c97f6895e0b1fb0ddbdcff83a563c26647664e4a7eae968c131e6a809e398c10ab5fbe118218dc00289e38bfe5520601c3c002
java.security.MessageDigest
157
not checked
si23.gif
local
1508762898824
0
SHA-512
a20335ae8a2f423dea04a3466f2f729faa4b2a4e77a359c2ca0dc25c59efbd64b8233fc30396c1d72f775c3cd7ff8a1d2d3077c2bab264406116650c1380af71
java.security.MessageDigest
200
not checked
si24.gif
local
1508762898825
0
SHA-512
da507529b8ff76626137ecd7ab50f107c904d47124f2644a05ba32c804cb4cab1562d50ce23093aaa664feb9903c6fc408a2908ed1fd46e7315a059d195a0719
java.security.MessageDigest
270
not checked
si25.gif
local
1508762898826
0
SHA-512
c09e505573271f16f45430da60ee8f6ad851bd92e726aad8e6223a5df853fdd3d26b5761ced6b43a9d2b6678c273d3791a1c3ec024fbe7c634bb59f5ddb832ef
java.security.MessageDigest
173
not checked
si26.gif
local
1508762898827
0
SHA-512
3d9414d35169de895e4a909d25ead95f5323b426323ffc28436ce046bf1cd28a7c5db5359fbda71ed0e09eb50cf6bdf95e90edc2d7c49209c446e5c11ac29dc2
java.security.MessageDigest
248
not checked
si27.gif
local
1508762898828
0
SHA-512
11e20040a2ea5153c3c40865666de903c3600ed1fa981b77128a97e1495b1cf5f3ca6363865f94a9275b492982f1bf980477ee517f2d84d9294fe09bfa565631
java.security.MessageDigest
229
not checked
si28.gif
local
1508762898829
0
SHA-512
0e146ac3ab0bc96f604fb8155ad1844e9a42c54bbf304ebbcddac2b775afbc9aad32b2705d3d13626840b548c96a0b5d01b4b44f97334f2894f60ebef8204060
java.security.MessageDigest
187
not checked
si29.gif
local
1508762898830
0
SHA-512
e48bb56dd3824eeeba9b5fa9f99370ac58a6e8e28e2a5e123f4cc66fabe8b2f438aec375d32750695642fb033da4ef668ac3317d1d9202b517fff655b788fc67
java.security.MessageDigest
117
not checked
si3.gif
local
1508762898831
0
SHA-512
b0b703cd3e092dae669b00f19b06b1efb47c44c4e139ed131366025413b388f9afeec115b769b021abc77aefe9eba5c1514f86735ded868e92244bae7c30c8b1
java.security.MessageDigest
182
not checked
si30.gif
local
1508762898832
0
SHA-512
cbd3900448927465680c36fe11f1eb904a44522a5b7aee42971332a0ee6aec56ae125b7a476a14ff278666b5d7b9ca6039d335e58470e64ce5f2a9521ad8cb3b
java.security.MessageDigest
205
not checked
si31.gif
local
1508762898833
0
SHA-512
e3de6b538f1ed53c5fc67395c6ea5cab93e32d177ca3adcd9fd51563cea94a2f6def066164ae14db4fc4a6eff35b82d735b848ab5803a8a4a51414bf95629594
java.security.MessageDigest
254
not checked
si32.gif
local
1508762898834
0
SHA-512
31275179e513430b104282befe94595272040d5dcf74f8ba3a559fbf72466fb694347b57ab9a17da3982f930191fdbefbbf38d602e1d618c6e4257fb900b8259
java.security.MessageDigest
200
not checked
si33.gif
local
1508762898835
0
SHA-512
7f25c895380e021b20aca3f03d5bb14df6971d35416acd67733f865f79c03c1cd614f4256d77b90b1f7ed6046f02e9b95bae3a4b7efa18fba81bcd700e3e8f8c
java.security.MessageDigest
437
not checked
si34.gif
local
1508762898836
0
SHA-512
c0991c70b0756ff059b255511da4badd322657c8a5bd454b5f075f37390e29d4effa3f3ff0b3a8ed456674125e882552a09860f1c5c97eee620a2135c78cb4eb
java.security.MessageDigest
340
not checked
si35.gif
local
1508762898837
0
SHA-512
7221dea8a91281e53918057d947632ed0e6c30d672d7bc12b94d711849596eeb5aae3ba6783c40f29a7ac111d10354dbf0874d1591782be8c713ac3fa735e402
java.security.MessageDigest
597
not checked
si36.gif
local
1508762898838
0
SHA-512
c3b62c6ca9ee5dbd5498aeb71aaef3f1b5d1b4c697304534bcc0ae9b4b9311918320c8bd15b70a9c3b884814a88818d194689f587a89bb35751324ae379a9109
java.security.MessageDigest
486
not checked
si37.gif
local
1508762898839
0
SHA-512
1e5773d77938eaf9d12e85c496c7ef0a41c05faf20eb31a6f06aea28e235075cffd582010ff2d6dad13a3a247a7bb88e363b42f43f506d1340cc8f912c24dd76
java.security.MessageDigest
161
not checked
si38.gif
local
1508762898840
0
SHA-512
ad2293503f221ca80c1290059c6e1533c487a5a835514d9535c64e9bb75535ad3adf1c7d1a35f87552d160b8a628a949d29bf68a2eb151693f3771337781c060
java.security.MessageDigest
343
not checked
si39.gif
local
1508762898841
0
SHA-512
eb266d7f66c35375caaae866675798ef02b7b002270a03169a75ce74df7ce64ae6b26c50bcd84e97671480ba05055310e094a7528b44f13ba39307d5583fe692
java.security.MessageDigest
159
not checked
si4.gif
local
1508762898842
0
SHA-512
fd12fe37f87d1b378c65931da5cca9a6a92b2528e6e511da7f2b538dfe5795d4c16b3111a155826909fc1876e03da89b7dbc574e340f519b439dd14f0d66eb4e
java.security.MessageDigest
195
not checked
si40.gif
local
1508762898843
0
SHA-512
2b5d22422567b706e6336ec7f97fb0bc6f88eaeeff8c687e76d96f714d84d4a306e1709c68171c51580e187f9af8094bbf73a3761e132ff0b0d47d6920ce74a6
java.security.MessageDigest
241
not checked
si41.gif
local
1508762898844
0
SHA-512
27e382050acb4461a385d8d3668a1bd0fa9c20c46b7dcca53394233262aa29de7b11a20661a1a4f7ed48341d2990a98bb5cd89697df0261e89b05dd9c91af4bb
java.security.MessageDigest
569
not checked
si42.gif
local
1508762898845
0
SHA-512
7cdd3c3d06915b1e873f6b42c217622e6a47776ba1dc9fb714f3512c75a9b92ef872810b672264617278ad27dd997c5d2da50b71d2a7d295eac1c31c1e551370
java.security.MessageDigest
647
not checked
si43.gif
local
1508762898846
0
SHA-512
d451d16aeb0841927138b3d977b2a01bd14af2518712f2a46001324953a6ff352e365205bbd8c089203e03aa12a54050e7c138f063f49e7d347032ae57a2966d
java.security.MessageDigest
442
not checked
si44.gif
local
1508762898847
0
SHA-512
98036e2dc68f40bbc39b64a704e5f996aad0ee4142d0b1045be5f3ee06bcd87570e113920155ae4954d4dd6078ff740a09a5a90d4fa76cd509b68b4b97f4d2e2
java.security.MessageDigest
732
not checked
si45.gif
local
1508762898848
0
SHA-512
cf42e9eed9713f67d9a7b88f5689f9d8e85f4f280896ecbb7c48d30b6022c8a9fefec77d7451372d94ebd6994950c9e8ef8fd9b9eaf96236d06d3593867efe19
java.security.MessageDigest
395
not checked
si46.gif
local
1508762898849
0
SHA-512
d840311774658836c56fe62143716e39338d422bc3bed43480a2cea3e347735eff0c9d0dae270ca94813121ed89cc445ba835b179b4188317b2153f3d21244cc
java.security.MessageDigest
187
not checked
si47.gif
local
1508762898850
0
SHA-512
2f88928dc1c3a81781181b5ae88d462354ef8a5746f999369f443d68fdd3be22feeb9359cd51500aff782b26736b8ac2c5e606cce25b83649eb6da936003b9b5
java.security.MessageDigest
162
not checked
si48.gif
local
1508762898851
0
SHA-512
a60c2430e1f7963c2bd7b4a9c7dcf448eea9bba9946b51d578f84e85749c70cf8394ea888e86a9957052e6743d974ce7b076967f17abd8e8ec40b766e7aae0db
java.security.MessageDigest
203
not checked
si49.gif
local
1508762898852
0
SHA-512
c34e8b02a75d24ecb757785688763c4599e05a3be0ba9846ea524c75f7d2451668d34d89779d5f713f50c1aa500c074a085c2b3492ca2477c1f8018dd50d4f4a
java.security.MessageDigest
118
not checked
si5.gif
local
1508762898853
0
SHA-512
b837a079c6a7c3d753845f06fbe410da7b02636e70bac9b27289060e442bacb98182ea67465ec3543fa1f0954f1dfc023ec967fff62a56e3a4e3976752918eef
java.security.MessageDigest
203
not checked
si50.gif
local
1508762898854
0
SHA-512
aacf7f4f26de6c005bd6af9c7c166081d24088a2d95ae0bee4e269d2806673eab38c71c338d7f146d69183c18b36f5b4f811ba2c3c39018e4a7a7dc5c936e421
java.security.MessageDigest
459
not checked
si51.gif
local
1508762898855
0
SHA-512
1890755712e821ab0436da414b31fd574f14a2509f71e370b4385eca79bf2060adfc6282b56d76c4816f16bfc0b143c175370d148e9d71cb1eb0b6f2bfa3ac6b
java.security.MessageDigest
212
not checked
si52.gif
local
1508762898856
0
SHA-512
4ff6d2778fc81c1561c7aa267bcda28478774d757013f1db9cf0c8649a5709d6ff9682844166f60dfd19eeef0ae114ab5c36d162d7f34c4f9188550ecacc30e7
java.security.MessageDigest
246
not checked
si53.gif
local
1508762898857
0
SHA-512
b4635a6e7fa7521173d946bd917d04ee932d4d2d5f86a1d38f6fd40c584687773a193e7890ffd24d87a01246f6bad1c1a900d5c36a696802e2fa5a462c7dedea
java.security.MessageDigest
717
not checked
si54.gif
local
1508762898858
0
SHA-512
f28f306dec2a2b101f536b043e028dd046e7415256f85e86a2086c03559f7f173913df11d80e683e6f87cc174e3767d43fb23ebcb04512a2552eb4378a288d88
java.security.MessageDigest
257
not checked
si55.gif
local
1508762898859
0
SHA-512
2fd60ac9898029d08749fd838a57bbf6afa4a80799135b38bc8b16a218ec230a99e31ed9d31bf6fdc0c9d513f8531263567b237ee603209a2d8f389849a85950
java.security.MessageDigest
586
not checked
si56.gif
local
1508762898860
0
SHA-512
0678737ed53ce5da20d00bf81076f86a30178efcc2633140d0e99f3c632618702a1977b4d5e925f04bbff0581b4471e0d8896dfd4d212f70816695e2dcbdd4c8
java.security.MessageDigest
323
not checked
si57.gif
local
1508762898861
0
SHA-512
e4d52a39e41b8681cf8ee2d26431c00cc967eee00fbcf6be92cd99510c2a456b866fa362c54fb8bfec4d60294c1526d45e377d518d3d95c6f7086a6fc476e984
java.security.MessageDigest
297
not checked
si58.gif
local
1508762898862
0
SHA-512
52c68f2bc7dbbc2af913e9de81911a1d5db596be68be35f51e53aa785a2196b8e38e957e9adc488980a3769f14804972e7628114dca48fa5f228dd7899dff653
java.security.MessageDigest
477
not checked
si59.gif
local
1508762898863
0
SHA-512
169c9d39942e8a29f12d565ece528766984d9674f1d3fef2a390251336f5043a6d68f79401012a02a780a5a774efa955744f430df8c253d3e88953b277c5e998
java.security.MessageDigest
678
not checked
si6.gif
local
1508762898864
0
SHA-512
95606663363da7bcdb137fdbc772a0aaeadd01ed523c21d03d7fb1aa12893131a040b6997cb7105009c44022ea0bcec5f4f7d2cd2425f92c6f57accd4116d8e5
java.security.MessageDigest
622
not checked
si60.gif
local
1508762898865
0
SHA-512
6a8d2ec2201fb2441094af2f62a3785ca1f76eac98530efa314fd34e89c2a8b8dfd42302a50ba5fbf102f4ea1e3962a8d288abd6d3c437b1d971b460c4670e58
java.security.MessageDigest
460
not checked
si61.gif
local
1508762898866
0
SHA-512
0aba672e3c8d4b088dab4fe6f0963976e4793b418c68091e04971c7286c77f68f6994444f982b8a5d9c9a96a37a42e81eb31e0f140b1c8ac05c1be2488415548
java.security.MessageDigest
204
not checked
si62.gif
local
1508762898867
0
SHA-512
eff06bd3de4004397878aea24a6bdb615cc7ba3c240b444e676dd439716ce85acb4322fc085278a356f5def7e5cf5b61fefa728314e20c7192f33d443ec462c8
java.security.MessageDigest
392
not checked
si63.gif
local
1508762898868
0
SHA-512
be1fba72b2a5cbd5ccb4b634e1a704144b0bc272d7a1abd419aff6a697ac30e46455ab7089c251bf0d7a72d6358acc2573ba500887bd5f31fc182744365b8809
java.security.MessageDigest
450
not checked
si64.gif
local
1508762898869
0
SHA-512
757b0280420fa0307e0a4ea2d87fdcee05df9d75dbe81db89f9f3062590bbf6d3714196e29ef6c51cf313aa1bb41fc567ec25a4cd151a46d819c1a3b7e280b0a
java.security.MessageDigest
145
not checked
si65.gif
local
1508762898870
0
SHA-512
0ccb37063dfe3f27f997ac434e1fe3260a9e11ba9fb709d1fa6521393de5b1c745e2687dade03ab434ddf2eb84d008ac1d108ec4ba3b389b17d81a3f3b686d55
java.security.MessageDigest
440
not checked
si66.gif
local
1508762898871
0
SHA-512
bb7574382ddcb3d7552d6211e2c76c36f649138337ff40fdca40025b9008ce8f4a65a9e921f46208fd00964bf1d131a5aa74ae72283a8190ef3c55c11cd34514
java.security.MessageDigest
148
not checked
si67.gif
local
1508762898872
0
SHA-512
1da5529007544e7ec7e4390549adc3dc7138bd4b2359e7be0b5365564c0bd94747efa274dedb9c05e315af65f33fe9f317e2e517a53127bc29ee3fb59b379bf6
java.security.MessageDigest
341
not checked
si68.gif
local
1508762898873
0
SHA-512
9290526516e082b41c096c607da9539a487f8587bf658e7a1d27e3ad943cb2860ccc57d17eb91f27c643e14810e7bc36c4ad45c40ec9164a9631cd1599c53dc9
java.security.MessageDigest
343
not checked
si69.gif
local
1508762898874
0
SHA-512
84124524cf314df75c67a96c56364eb7c40a963de0cb9648f6213b038134277adbb94cd575998759abfe4ee8be8aee37f6990f299018858a7237e2ced7f75e77
java.security.MessageDigest
180
not checked
si7.gif
local
1508762898875
0
SHA-512
1065af1e519d96b60f820650dd15d97fe51dd1bf47eac1ff9f01486c4dd2f033d5ea564bc27e98fa34c9fafa07a7eff8cdb66a079e87a45d01313b9354632847
java.security.MessageDigest
1016
not checked
si70.gif
local
1508762898876
0
SHA-512
fd5f7f42e2521c6fc0626caf8728d7d8bd4124e039d66c2a1a9c9171298acd18b008d04516d9bb232614d0994f45c456b819146733a1ba7147d1b0fcc3e45def
java.security.MessageDigest
1000
not checked
si71.gif
local
1508762898877
0
SHA-512
d092f74e35da6b8e617182f340099cf22dd5b39ce4d29599ec1ee1bfff7ee8134b8817795450a01417468440216bda0363efba402dc7ed1c7be9d2d93d068edf
java.security.MessageDigest
133
not checked
si72.gif
local
1508762898878
0
SHA-512
311e032a4ff9b88573fd6c705bc6d7684c664bcbf8fbc1705d06e1f64960dd154f03978deee7c03ec2c0a098faccc7fc04b686cd5a73fd43942ce5a2bdd37646
java.security.MessageDigest
140
not checked
si73.gif
local
1508762898879
0
SHA-512
9fc625f0bae1fb099e1467b59ba14cdf579fc73863db9bdc005709b7c0ec65a7bea02cadee9ce2f7e737c52f76e4a0c854d4e016390efc89ef5e279f454d62e6
java.security.MessageDigest
799
not checked
si74.gif
local
1508762898880
0
SHA-512
9ef5934918718ce510df30b4397de0523d54d3b51b2a549ea74e4f20bbf32021cee56b8f0b450af2e2d0ef6e529e80f40d749e6424d9ffd2adb2ef9e21d077d3
java.security.MessageDigest
981
not checked
si75.gif
local
1508762898881
0
SHA-512
67add5bc68f59635449980f785b0582b7b7b727d5457a0f4cb4a33e46a8563d663138c4d93de63c03e41513774fc9c9f0d845ffcff77fc68fc5243feb70e0cc2
java.security.MessageDigest
946
not checked
si76.gif
local
1508762898882
0
SHA-512
5ec4da29f7d8f78cb7a98486d14ff096b273d4efd2c2e49750fd8c21fbcb0b954e2bc232d56f1a17d4d42176c07fd2353f34632e4913e35896601b660409d7a8
java.security.MessageDigest
810
not checked
si77.gif
local
1508762898883
0
SHA-512
d4581d57f6df08d541d9636cb02384c5964d7a57883aa96f20848db9572ace09b0e17262ad6f91a1830ae96022d34967d381d246b6ee066cf73c31784b4f8284
java.security.MessageDigest
346
not checked
si78.gif
local
1508762898884
0
SHA-512
0780d6b5549b87af156f27cd4f036e3a0efdaefd5f9ce16ecd4f34cc4413cf8e6d7f10f4c8a26587a439a34b54603b0350b8e0f0dc1d64bae65efec2fdeae720
java.security.MessageDigest
247
not checked
si79.gif
local
1508762898885
0
SHA-512
251299fc59ca3b5560a17dc58a6be92b3e45bbe38fbec3f7fd822d670a72517e9f037d715c9219a352ff0865a1364427bee9aec751e53931d312405038b2e8e6
java.security.MessageDigest
117
not checked
si8.gif
local
1508762898886
0
SHA-512
c855bab2b67e7e9df9598b0c5c29a9d3b2be8c55d12e47893957e359fc1dde0ff867bc0d4171ebbc26200e4a988075996716de535506e240649f0db6678b2e4c
java.security.MessageDigest
312
not checked
si80.gif
local
1508762898887
0
SHA-512
454c699c323360a80b8ab33989b788e769d9751c9a7d8d7fb2f5525bf4340d6a5a9433fbdcf243179e6db1ff89f1b4d6957c0db8a0affd3e4c9e229595db738f
java.security.MessageDigest
379
not checked
si81.gif
local
1508762898888
0
SHA-512
6ca4c2f07d62ff146aa689c04f7de27cf2a6d0f55ae6c3d59aa10c032beaff07e2f28c7815dbec8e8a863d7c73840e42acf6272d8dd8983bd87f9b3555e50867
java.security.MessageDigest
382
not checked
si82.gif
local
1508762898889
0
SHA-512
333475688c938a727232b37a9d6527d529c83f250ea3a20d9c246e8f0b30e88f2ca9b208f408e12230d8adde3086505fc1ab2c55c9b2292b815d3121f4e4eb91
java.security.MessageDigest
1903
not checked
si83.gif
local
1508762898890
0
SHA-512
1f50343acc2654a846dff9d38d0129c26a9975f520759813eb3648c2b64a741261f6a29a87b171553a287d4dc1adbcfe312360d3568d23534ddfc561cbfaf745
java.security.MessageDigest
142
not checked
si84.gif
local
1508762898891
0
SHA-512
3a73351d1869024975329fd60568b03af1e7cf970e2c426c895f0c2ef3e8b1b6d39ae33d4cca76cf66d9d4005bbae197fec81199c3cee8a0249a2c195023e682
java.security.MessageDigest
245
not checked
si85.gif
local
1508762898892
0
SHA-512
cf7564879b5cadd7042008e6eaa194660639acf8f4db32cc6c26f66412a284fe0a5ad7c1a7b670722920ebe912eb918158b5e39aaf71b3b76d435f11cc60ddd3
java.security.MessageDigest
243
not checked
si86.gif
local
1508762898893
0
SHA-512
b84f52f31a146c7db1cde5f228f7ad66f06bd1c8e5137cdd9b0d0bb86a7db9d1bb7dd7f79bb2a813a689d5f8d8bd32f6f16ad2a7b25067b4bea49aa2e9f05e53
java.security.MessageDigest
333
not checked
si87.gif
local
1508762898894
0
SHA-512
7ebaba7ea1bfe606287bdccd39641ac27702f930d61f0a6622f34bf00a85e0fcede9c3fa81a5a44277a65109d17a27fc8f1357a944293cb1ce91a63bda548772
java.security.MessageDigest
648
not checked
si88.gif
local
1508762898895
0
SHA-512
20cb02d0cda6a239137de0aea9303021b68fc3ac1462c85e98e4adb90f7636b7db694e63848ff73ce0f9e278e914ebe0a12d03005bd6da14e561027a891e2fc4
java.security.MessageDigest
136
not checked
si89.gif
local
1508762898896
0
SHA-512
76ab4548bff814bcacbc85f447928061724d345bbe7fd82cb49535bdf0e710436bd8128d1f715f0900a0b8d084aaf962878f4f1471abf6c1a3e6784da880120f
java.security.MessageDigest
126
not checked
si9.gif
local
1508762898897
0
SHA-512
e6c2935f3ef892ba97f713c018b729fb4954bd9b5d7cbbf3a54d80fbfd65fa58c4dd11be85e5c40346f76d6e6dbe7c704685e25755eaa386d2e60cd1b3107722
java.security.MessageDigest
365
not checked
si90.gif
local
1508762898898
0
SHA-512
a59465fd5121d6334c89c239cd4fedbec4496bdb16ce80bb85927105dfa39ef941e23790e92520ccb6bd19e1969a2a0f2e7277570ba3fd75e2daec4ddcbb00e8
java.security.MessageDigest
240
not checked
si91.gif
local
1508762898899
0
SHA-512
8e309caa4f6bb8a9fb270092985df84bfc7ef01eaca9ac777f124bebb07f2b4a13eb788cfaa053460f349ee6e9c55e99bf845106b60c323d8fcd0f7db8c71fe3
java.security.MessageDigest
138
not checked
si92.gif
local
1508762898900
0
SHA-512
3558622e8c14e5333b0d39841b1192adab5e9bdfd5b525470192838bf264a36c15bdb129795df4e85c16c4ae0807519e13498e36dcf5403727b7db34f359cb80
java.security.MessageDigest
147
not checked
si93.gif
local
1508762898901
0
SHA-512
f8f0a5ec66093fc6f3371aeb548497cf9bc70e1e6bc529544f039ab21fe5cd627a476d82101f7a4d899d3c8d73d8decabf801dde899cb7e417bcba99e5bfddb3
java.security.MessageDigest
379
not checked
si94.gif
local
1508762898902
0
SHA-512
73cdac657639945a24ba3e121c4baab3b7ca0f95e08dc93faeb7021b6c9ccb21e16c4c788b5d38db4ab7f556666c0559f6bba95586d1c673d463c50de228e364
java.security.MessageDigest
253
not checked
si95.gif
local
1508762898903
0
SHA-512
aaa7d178c121c395d5763c016608068b3dc535fcb987affe669e5f62e5f96dfecf0d56453febce52a9af567e77ccd243934623c904f7951378260e1635009382
java.security.MessageDigest
389
not checked
si96.gif
local
1508762898904
0
SHA-512
81aaf3d8b16b5a1462579b956ce359763872915f6c9da153b77a9d24c09d49a4a14e29213c373e913eaa6c4ed49cd44939d51a6387442dbe58d51930d733a373
java.security.MessageDigest
171
not checked
si97.gif
local
1508762898905
0
SHA-512
dab61e8dec45272e2b3e1890cd37a9f22e154cd395d39a3f2550ef97165d9b9c0c5a2035890cb5078dc0c173e761a861f5d8c7b0ceeb8e8db49b8b7762173d54
java.security.MessageDigest
330
not checked
si98.gif
local
1508762898906
0
SHA-512
eb6b749c5a8b9d03718428d0342430e1a38c9f3c9eb3238a5717751491264f1ab35c8278054008d73d66213b1c21074ce0cb41be57195c805797195d8046e3af
java.security.MessageDigest
574
not checked
si99.gif
local
1508762898907
0
SHA-512
108c5c5c332255147e5fff848a044637876ef1328e42670b7ebbab9c3c0bf27ca6a3d673098f0ad8934af2975ca863b557985d3d86a65e7cbb2062af722adf8b
java.security.MessageDigest
46700
not checked
fx1.jpg
local
1508762898908
0
SHA-512
31e3c683e34e73089ac991c1ba3287585814770729e4e6ccf6e3096d31620886e3f90064e3948dd75437b370996a734e3bf413939a1749d5646b99a9855aea2c
java.security.MessageDigest
13520
not checked
fx1.sml
local
1508762898909
0
SHA-512
10b1f87daaf591454f3900e359de9c6bde84c9d3d4006b872e1e581505f7cf309affa889805f5f7b47c96f6ddfa819bec2ac9f1230235a391fecdd0ab8ad93fe
java.security.MessageDigest
88436
not checked
gr1.jpg
local
1508762898910
0
SHA-512
25e72b9863b04e3b2d7f542f2ac8ba2216865c65bea0c93be1b75d298331519518d21e0e0dc134baf58e17e9473a5bd38fe44094890ad4dbdf179b2d773806d1
java.security.MessageDigest
14286
not checked
gr1.sml
local
1508762898911
0
SHA-512
10b52a6665e81e75711f028f3ba00e5ec65c1d9f582f90a2e64bd5f3baadfd089f7b0de11551a4bb82da3062032a8b9a7232de0fd465e82ba740427c22f86dff
java.security.MessageDigest
107597
not checked
gr2.jpg
local
1508762898912
0
SHA-512
4f68d8761fa4fcc11a5b16dd7d25e0cc16b401e82c282e6be7cf3089f3675d28215c90b358f37ff0a2fb3f15b612bf3c1967261ad03d4dc05369e510dbb958c9
java.security.MessageDigest
13710
not checked
gr2.sml
local
1508762898913
0
SHA-512
c82b22aa01c3f2fdbad4c315d242ce6ed4175b10df191125e9fafd39c2f4b903aee9097e2dabaadabd569895d0ef2d0b9224231b7bbda08338c09ae535d833f8
java.security.MessageDigest
88819
not checked
gr3.jpg
local
1508762898914
0
SHA-512
647836896462889f5e79448458388df5872bdbf582dbde992c41be0cd0a79855d0087a60342e016abbe329651b19316fc797c8521678084f530f39d4f6aa5790
java.security.MessageDigest
12418
not checked
gr3.sml
local
1508762898915
0
SHA-512
a0901972141e18ceddfefc5b26fc4e5d5e2f73e5b7679fd53bee678326ca94ae2a82071c5c90e5548d80ad7898a78f687cc69e6498d42e552c9f59d0e4d583e3
java.security.MessageDigest
98075
not checked
gr4.jpg
local
1508762898916
0
SHA-512
95db61cbacb6284af4a0cd505720f34e279d8e1fd22eb64635fc7cfed219f1945552a7dfb19c848d6806f49d8adb40f54dc559f4758abcd4ccd2cb21aa31873f
java.security.MessageDigest
12830
not checked
gr4.sml
local
1508762898917
0
SHA-512
f29bbea4c655de12ba4dba08984bdda283c2328d4c13fb26c7e301394fa07d0790bc4f0cd82317695634595ac056d4ccc4167ad35e3860705ed3b902330aea2f
java.security.MessageDigest
108484
not checked
gr5.jpg
local
1508762898918
0
SHA-512
c1cf0040a7f9713738f683a4092dd5b9591a50b9c036cccde99573f0eccee4e4847c72820aeec16bd67e78984376489a660ad1ed939625702fd50469bf42c504
java.security.MessageDigest
13743
not checked
gr5.sml
local
1508762898919
0
SHA-512
a19df5202e273a5fb65e74e51ed52291d30e3a6ab8af3764d408bcbb813e0be6607c7390a4cd6dd92997b5a82bad5ef8693bb319ba8f6f330af5aee2dd95c963
java.security.MessageDigest
68436
not checked
mmc1.docx
local
1508762898920
0
SHA-512
e8414fc28137f16789958f906c7860b3d36ab827c2fb7ec59d003c27cc7b6f51cf51d0919d5fa72add84667bc354cce13d70b7b164ef4f0ef5c36b474ad41d0a
java.security.MessageDigest
145149
not checked
mmc2.docx
local
1508762898921
0
SHA-512
e2e366f5e5aeba812ecea20cb16766aa3ddcfae838c9c3eaf711ff861573986b814cb1b578ed8b9b1b084ec5a33f8dd746eb0c7f87eb18f46c6158675f067df7
java.security.MessageDigest
2928016
not checked
mmc3.docx
local
1508762898922
0
SHA-512
a5cf5607b790352d425340cf18fe7161dec42f58c0efbdf068048c2222f76615eeb824e05392ea24554286c8f16737f1b95974eda472b16344b66ba9b1dd6fc9
java.security.MessageDigest
3494492
not checked
mmc4.zip
local
1508762898923
0
SHA-512
e8ac0f3db00d92c3b492606c0e1ab7d67498ef57d9d3b08c967a6746d746db099c46e08ea8c50cda84e141f95ee162af434b098fc469baf11771169fb1db0fd0
java.security.MessageDigest
16462
not checked
metadata.xml
free
00001
The Authors
KB-agent-id
1
supplier
KB-owner-id
00001
KB-agent-id
1
Elsevier
organization
The Authors
ingestion
2017-10-23T12:16:43.308+02:00
Connector
software
Digitaal Magazijn release 1.5
ejournals_esp_1
streamprofile
ingestion2017-10-23T15:27:03.740+01:00Generic IngestsoftwareDigitaal Magazijn release 1.5