Advanced-EFA (Hyball)
Download: Advanced-EFA (Sept 18, 2003)
W. W. Rozeboom
September, 2003
GENDATA: A program package for creating artificial data with
complex factor structure
Preamble: Why simulate?
a) Create datasets with idealities or special features wanted for
educating and testing students, or for studying the comparative
accuracies of alternative solution methods.
b) Test for ease or difficulty in solving for particular hypothesized
structures, especially when the ideal is contaminated by sampling
noise or other model violations. In particular, which parameters
in a favored model are most/least robustly recoverable under varied
degrees of misspecification elsewhere in the model?
c) Explore alternatives for parts of a model, intended for fit to real
data, that are convenient guesses rather than hypotheses supported
by what is already known or plausibly surmised about the model's
intended real-data application. (Even if you don't actually create
any Gendata simulations, consider grounding some student seminar
discussion on the options it avails for choosing a model's exo-roots
structure and how plausibly these should occur in reality.)
d) [More? ÄÄ suggestions welcome.]
ÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍÍ
Below is a packing list, with brief overviews, of files in the Gendata
package. These are of four sorts: (a) executables and the Fortran-90
source code from which they have been compiled; (b) major or minor textfiles
documenting these programs' usage; (c) several programs from my EFA package
(Hyball) that do things with suitably formatted rawdata files; and (d) some
textfiles containing unpublished though circulated views/proposals on certain
controversial aspects of SEM practice.
However, the descriptions here won't give you much operational sense
of how to run these programs and what they can do for you. So skim this
document only briefly before turning to SIMDATA.TXT for your main instruction
in use of this package. Afterward, you can reread selections here with
rather more appreciation of how the components they describe constitute
this simulation procedure.
Gendata's programs for creation of artificial data were initiated some
years ago for my personal study of solving data covariances for common
sources that produce factored data in accord with the premises of classical
factor analysis but with patterns of factor weights in model Y = WùF + E
considerably more complex than the nearly degenerate independent-clusters
ideal that has dominated past EFA simulation studies. Gendata's original
production resources ÄÄ in retrospect, rather narrowly conceived ÄÄ are
still included here but have been much expanded to provide feasible creation
of simulation data having almost any structural complexity that modern SEM
analysis might hope to recover from data covariances. Textfile SIMDATA.TXT
in this package describes in considerable detail the models that can be
simulated, followed by operational instructions how to do so once you have
completed the installation.
All programs in this package run under operating system DOS, either
stand-alone or in a DOS window, and should be kept together in a dedicated
subdirectory whose name you will later want to put on your DOS searchpath if
you decide to make serious use of them. The output files they write will
collect in whatever directory is active when you run them, and if you create
data for a series of projects you will probably want to spread that among
assorted subdirectories different from the one containing the production
code. (For initial acquainting yourself with these programs, however, you
may prefer to install them in a temporary subdirectory and test-run them
there.) The only files in this package that actually do something have
names of form .EXE. These are in binary code ÄÄ you can't read
them in a text editor ÄÄ and are activated by typing at the DOS
prompt. (Don't be shy about starting any of these before you're clear on
what it does: Each gives an initial on-screen overview of what lies ahead,
and you can safely abort the run by hitting Ctrl-C whenever you prefer not
to continue.) The counterpart .FOR files are the Fortran-90
ASCII source code from which these EXE-files have been compiled: You don't
need these, but they are included in case you plan to do some simulation
programming yourself and find ideas or procedures in them that you can use.
Also they contain internal documention, my personal reminders, that might
clarify an occasional uncertainty about the programs' operations. In
particular, all the runtime screen displays can be read there.
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
A. Here are the Gendata executables, listed in roughly decreasing order of
importance with name-extensions omitted.
GENSCOR: This produces an ASCII matrix of standardized covariances (corre-
lations) Cyy for NV data variables Y having NF common factors (sources) F,
and unique residual influences E having classical orthogonality properties,
that give rise to the Y-scores under a production model Y = WùF + E whose
source pattern W and source-correlations Cff are stipulated by you up to
scaling adjustments required to standardize Y. At your option, GENSCOR
also writes a file of joint scores on satisfying this stipulated
composition for NS simulated subjects whose size (NS) and, if wanted,
nonNormalities of distributional skew and kurtosis, are for you to choose.
Before running GENSCOR, however, you must first prepare pattern W at least
schematically using program SCHEMAS or possibly GENPAT. The same is
largely true of source correlations Cff except that your three alternatives
for how to prepare these, detailed in SIMDATA.TXT, are more diverse than
are preparations of W. With one qualification, GENSCOR's output of data
covariances Cyy is Gendata's primary end-product, ready for upload by some
other multivariate program/procedure for whatever purpose has motivated
this simulation. (The qualification is that you can also take Cyy to be
Cff in a multistage Genscor simulation.) The scorefiles *.POP produced
by GENSCOR are likewise Gendata end-products except that further Gendata-
availed operations on these will be wanted before you put their simulation
scores to external use.
SEMCOV: For simulation of data manifesting the sorts of causal-path
dependencies among common sources F that modern structural modelling
(SEM) seeks to recover from F-indicator covariances, SEMCOV computes the
F-covariances Cff entailed by a posited path structure A conjoined with
the posited covariances Cuu among exogenous inputs to the endogenous F.
A schema of path matrix A must have been previously created by a run of
SCHEMAS, while Cuu has the same three preparation alternatives that Cff
has in GENSCOR runs. SEMCOV's output Cff has basename of form KOV* and
is intended primarily for upload by GENSCOR to affix a pattern W of F's
measurement manfestations.
Unlike SEM models, EFA simulations generally impose no causal-path
dependencies on their manifest variables' F-sources and hence make no
use of SEMCOV. (EFA by Hyblock is an exception.)
GENPAT: This is an alternative source of common-factor measurement patterns
W for input to GENSCOR. Its patterns specialize in complex block layouts
that SEM-oriented simulators will find distasteful albeit challenging. And
its output filenames (or "vilenames", as an overly frank typing error has
brought to my attention) are near-intolerably ugly. (They encode rather
more information than really needed there.) You can safely ignore GENPAT
at outset of your encounters with Gendata.
SCHEMAS: This program lets you build a library of schemata for pattern
and covariance matrices for SEMCOV and GENSCOR to upload. SIMDATA.TXT
explains these in detail; here, it suffices to note: (a) These are quite
easy to create and revise. (b) Each comprises a grid within which nonzero
elements are flagged by placeholders (letters) that go proxy for numeric
values or randomization ranges specified in an assignment table that can
be revised when the schema is uploaded. And (c), when a run of SEMCOV or
GENSCOR needs an input matrix, it lists the available schemata having the
size and character wanted and allows you to browse for your preference if
filename alone doesn't sufficiently prompt your recall of that.
MORSCOR: In order to maximize retained information in minimal storage space,
each scorefile .POP written by GENSCOR is in binary code that
will presumably need translation into ASCII if you want to work with this
score distribution. MORSCOR does this for you under your choice on options
that include partitioning the uploaded POP-file's score records into groups
that can be assigned different means on their source variables to yield
modified data suited to study of multiple-group analysis. MORSCOR can
also reconstruct the LOG-information written in ASCII by GENSCOR when
it produced this POP-file in case you have lost its original archive or
want this with 3-decimal accuracy.
SAMPLCOV: This provides material for studying the effect of sampling noise
on SEM source recovery. Using the Odell & Feiveson (J. Amer. Stat. Assn.,
l966) algorithm for simulating covariances in random samples of stipulated
size from an infinite population whose to-be-sampled covariances are given,
SAMPLCOV creates simulations of the X-covariances in K independently random
samples of size NS (your choice of K and NS) from an infinite population
wherein Cxx is what you stipulate. This is tantamount to creating a set
of K bootstrap samples of size NS from a POP-file generated by GENSCOR to
have exactly Cxx for its datascore covariances from untwisted production
axes. Should you want genuine bootstrap samples from a POP-file, you can
run that through Hydata-supplement program HYDATA (described below) to use
the bootstrap option it proffers when computing the input datafile's
covariances.
COVFORM: Run this to set a small parameter file named KOVFMT that instructs
GENSCOR how best to format its ASCII data-covariance output for transport
to the solution program you want to receive this.
ORDER: This clarifies the structure of a disjointedly assembled causal-path
structure or, more generally, of any directed but possibly nonrecursive
graph whose links, described piecemeal, needn't be allegedly causal.
Operating upon a set of provisional numeric indices (they can be assigned
randomly) for nodes in a causal-path model, program ORDER (a) receives
keyboard entry of a list of index pairs signifying that node i is
directly path-antecedent to node j; (b) sorts these indices into disjoint
"units", each of which is either a singlet (strongly recommended if this
is a SEM's intended path structure) or comprises a group of nodes wherein
each node is path-antecedent to all others in its group; (c) identifies
one or more sequences whose left-to-right order embeds the strict partial
order your input pairings impose on these units; (d) clarifies the degrees
of path-separation among these units; and (e) for each unit that contains
multiple nodes, identifies all the closed loops within this unit as well
as each member node's counts of direct input/output links to nodes outside
this unit. This information enables you to re-index these nodes in closest
correspondence to their degree of path-dependency and, if it reveals loops
you did not intend, makes perspicuous where you can break those most
parsimoniously.
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
B. These are the Gendata documentation files in decreasing order of
importance.
SIMDATA.TXT: This describes in considerable detail each Gendata program's
capabilities, how to invoke those when your simulation wants them, and
how to plan your sequence of program calls and uploads. And it begins
with a comprehensive overview of linear structural modelling, both
classical and modern, that you may find conceptually instructive even
if you never create any simulation material using Gendata resources.
READ_1ST.TXT: What you're now reading. In addition to providing your
initial overview of Gendata, it will continue to avail look-up for
which filename to call for doing what job.
HYSTAND.TXT: This contains excerpts from Hyball's README.TXT documentation
that explain the use of certain Hyball programs, included in this package
and briefly described below, that carry out assorted operations on suitably
formatted rawdata files. Unless you do creative things with the simulation
datafiles that GENSCOR affords, you will probably have little use for these.
COVFMT.TXT: This describes the layout an ASCII file requires to transport a
covariance matrix for upload by Hyball/Gendata programs that operate upon
those. All transport covariance files written by programs in this package
have this layout, but you may have occasion to initiate some outside of
that process or do some editing thereof between production and upload.
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
C. These are the Hydata-supplement programs included in this package.
HYDATA: This reads ASCII scorefiles in many alternative formats from
outside sources, does preprocessing as needed and, if that is successful,
allows one or both of two outputs: (1) Transcription of the really-raw
scores into an ASCII Hydata-standard format that HYDATA and all other
programs in the Hyball package that work on raw data can read without
preprocessing. (2) Computation of the standardized covariances
(correlations) among the input variables or any selected subset thereof,
written to a COV-file that any Hyball and now Gendata program that
uploads covariances can read. (This computation also avails search for
possible quadratic relations among the data variables, and production
of COV for each of up to 156 bootstrap samples of the input data.)
RESCORE: This allows computation of scores on many user-stipulated
functions of the variables in an uploaded Hydata-standard datafile.
The newly created scores can be appended to the input datafile or
recorded in a separate file. RESCORE also allows input of an
externally prepared M-by-N matrix of coefficients that will generate
scores on M linear combinations of any selection of N variables in
the input datafile. Details of how to prepare this in a text editor,
not at present included in RESCORE's documentation, are available on
request.
SELECT: This copies selected subsets of variables and/or records in a
Hydata-standard datafile to a separate D-file.
MERGE: This enables two or more Hydata-standard datafiles having some
(not necessarily all) named variables and/or record IDs in common to
be combined into a single datafile. The input files are merged in a
user-chosen sequence whereby record IDi's score on variable Yj in the
merged file is the last score that occurs with coordinates in
the merge sequence.
An example of what can be done with these resources is revising the
scores on a subset X of the variables in datafile .D1: First, select
scores on X into a new datafile .D2. Next, use RESCORE to transform
.D2 as wanted. (E.g, the selected scores could be rescaled to have
different means and SDs, or partitioned into categories, or nonlinearly
reshaped as by Log-transform.) Finally, merging { .D1, .D2 }
in that order overwrites the X-scores in .D1 with their modified
values in .D2. And if .D2 contains only a subset of the records
in .D1, the merge will overwrite scores on those records only.
Hyball's documentation for these Hydata-supplement programs is excerpted
in this package's textfile HYSTAND.TXT.
ÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄÄ
D. Specialized background readings (contentious). All are in ASCII format.
PRELOOPS.TXT: An introduction to important details of scientific
regularities' conceptual composition that are mostly supressed, seldom
wisely and often obfuscatingly, in standard multivariate algebra.
LOOPS.TXT: My effort to make clear for SEMists who see nothing wrong with
nonrecursive models that these violate our most basic intuitions about
causal influence.
MORLOOPS.TXT: My 26/9/2001 Semnet post showing algebraically the special
conditions under which it is possible, under the standard linear model
of system dynamics, for indicators of source variables in a path loop
to have uncorrelated residual disturbances.
SMEPSHOW.TXT: A reduced version of my 1997 SMEP-meeting handout attempting
to demonstrate how EFA can effectively play in SEM's path-detection league
by use of the Hyball package's Hyblock procedure.
Wm. W. Rozeboom