Validation Suite
Home ] Up ] New Stuff ] Minix Port ] Magic-2? ] Overview ] Photo Gallery ] Construction ] Technical Info ] My Other Projects ] Links ]

1         Magic Architectural Verification Test Suite

1.1      Overview

The purpose of the test suite is to validate both the software simulation and  the physical implementation of the Magic-1 computer.  Some validation features, such as microcode coverage analysis, are applicable only when running the tests under the simulator, m1sim.  Because Magic-1 is not pipelined, instructions will be tested as individual units.  In practical terms, this greatly simplifies the verification process as it allows us to ignore sequences of instructions and the associated nasty combinatorial problems.  The tests are structured in a series of groups based on increasingly complicated functionality.  They should be run in order, as later tests rely on the correct behavior of instructions validated in earlier tests.

The tests are written in Magic-1 assembly language, with the caveat that the test files must first be run through a C preprocessor before assembling with qas, the Magic-1 assembler.  The primary purpose of the preprocessing is to insert in each test some boiler-plate code for startup and pass/fail reporting.  The test files are of the form <letter>_<3-digit number>.s.  For example, “d_002.s refers to the second test in test group d.  The output of the preprocessor will yield the same file name, but with an “sss” suffix - d_002.sss in the previous example. 

In general, the tests will follow a pass/fail convention based on leaving special values in fixed registers and then executing a halt instruction.  When running the simulator in test mode (-t flag), the simulator will print either PASS or FAIL to standard output, along with the test name.  The exception here is the first group of tests - those which validate the basic instructions required to execute the compare/pass/fail portions of subsequent tests.  Because they cannot assume that any portion of the implementation is correct, special constraints (number of instructions executed and values registers should contain) are passed to the simulator to check for correctness.  Also, when running in test made the simulator will enforce a “deadman switch” to detect , abort and fail a test which appears to be looping.

1.2      Tests

The root of the testing structure is the "tests" directory.  In it there are working directories for each of the test groups (though I'll probably delete these sometime soon).  To really run tests, go to the "all" directory.  There you'll find a perl script, "run.pl".  To run, do:

perl run.pl [tests] [options]

Where [tests] is a match string for the test names and [options] are compiler options.  There is also a special value for [tests] -> "all", which will run all tests.  Here are some examples:

perl run.pl b -t    /* run group b with test option (m1sim looks for pass value) */

perl run.pl all -t -c /* run all tests and show coverage stats for each indivual test */

perl run.pl bcde -t /* run groups b,c,d and e with test option */

There is also a directory called "stresstest" which contains a single test file which includes all of the individual tests into a single test.  Run using the "run" batch file you'll find in the directory.

Here are the tests:

Group A

Basic coverage for test boilerplate
a_001.s Simple br, cmpb.eq, cmpb.ne, copy a,b, load immediates
a_002.s Simple loads and stores, immediate comparisons
a_003.s Push/POP of a & b, pop pc (ret) and call <disp>
a_004.s Complete template code

Group B

Load immediate
b_001.s 8-bit immediates, no sign extensions
b_002.s 16-bit immediates
b_003.s 16-bit load immediate using 8-bit sign extended immediates

Group C

Register copy and lea
c_001.s Register to register copy
c_002.s lea, all 16-bit offset with SP, A, B, DP and PC.  Both a and b targets

Group D

Memory loads & ldclr (paging off)
d_001.s 8-bit memory loads
d_002.s 16-bit memory loads
d_003.s ldclr

Group E

Memory store & memcopy (paging off)
e_001.s 8-bit memory stores
e_002.s 16-bit memory stores
e_003.s memcopy
e_004.s memcopy

Group F

Push/Pop
f_001.s Push & pop (except for pop of MSW which changes control bits)

Group G

Add/adc
g_001.s 8-bit add
g_002.s 16-bit add
g_003.s 16-bit adc

Group H

Sub/sbc
h_001.s 8-bit sub
h_002.s 16-bit sub
h_003.s 16-bit sbc

Group I

and
i_001.s 8-bit and
i_002.s 16-bit and

Group J

or/xor
j_001.s 8-bit or
j_002.s 16-bit or
j_003.s 16-bit xor

Group I

shift left, shift right, variable shift left, variable shift right & sign extend
l_001.s 16-bit variable shift left/right & sex
l_002.s 16-bit shift left/right

Group M

Compare
m_001.s 8-bit compare
m_002.s 16-bit compare

Group O

Call, return & enter
o_001.s call, return, enter 8, enter 16

Group P

Branch on bit set/reset
p_001.s 8-bit branch on bit set/clear
p_002.s 16-bit branch on bit set/clear

Group Q

Unconditional branch and branch on condition
q_001.s Branch and branch on signed condition
q_002.s Branch on unsigned condition

Group R

Compare and brach on equal, not equal
r_001.s 8-bit cmpb.eq
r_002.s 8-bit cmpb.ne
r_003.s 16-bit cmpb.eq
r_004.s 16-bit comp.ne

Group S

Compare and branch on signed less than, less than or equal
s_001.s 8-bit cmpb.lt
s_002.s 8-bit cmpb.le
s_003.s 16-bit cmpb.lt
s_004.s 16-bit cmpb.le

Group T

Breakpoint/overflow traps, reti and syscall
t_001.s trapo & reti
t_002.s syscall & reti
t_003.s breakpoint & reti

Group U

Paging: set up page table and memory copies to and from system space
u_001.s Basic paging setup
u_002.s Copy from system to user space
u_003.s Copy from user to system space
u_004.s 16-bit load from code space and store to code space
u_005.s 8-bit  load from code space and store to code space

Group V

Supervisor/user mode switch, ssp tracking, priv instruction trap
v_001.s Set user mode, priv trap to supervisor, verify ssp tracking

Group W

Sram vs. device address space
w_001.s Sram addressing

Group X

Page faults, present and not writeable
x_001.s Page not present trap
x_002.s Page not writeable trap

 

1.3      Test Conventions

Each test should (with the exception of group a) should use the macros defined in tmacros.h.   The following macros should be used:

    TEST_INIT(group,num) -> group is the single char group name.  num is the test number.  This macro also provides definitions for some useful memory values (listed below).

    SUBTEST(number) -> Causes reinitialization of registers and copies number to a special memory location and register C.  Useful for quickly detecting which subtest failed.

    SYM(label) -> should be used to predefine all branch targets and other labels.  The macro must be invoked before any use of the label. It will redfine the name with a unique prefix based on the group letter and test number.  The purpose of this silliness is to allow simple concatenation of any combination of test files without fear of label duplication.

    END_TEST -> Typically expands to a branch to pass.  Should be the last statement in the test (with control falling into it if no failures have been detected).

When running under simulation, the MEM/ROM switch will be assumed to be in the MEM position, enabling 32kbytes of SRAM from 0x0000 to 0x7fff.  This memory will be filled with the pattern 0xa5, with the exception of a series of useful pre-loaded memory values that may be used by tests.  Note that tests should treat these values as read-only, and test should not make any assumptions about relative location or alignment of these values.  They are:

MEM0x00_8:         defb 0x00

MEM0x01_8:         defb 0x01

MEM0xFF_8:         defb 0xff

MEM0x0000_16:      defw 0x0000

MEM0x0001_16:      defb 0x0001

MEM0xFFFF_16       defw 0xffff

MEM0x7F_8:         defb 0x7f

MEM0x80_8:         defb 0x80

MEM0x7FFF_16       defb 0x7fff

MEM0x8000_16       defb 0x8000

The test framework reserves memory locations 0x0000 through 0x0100.  All other memory locations are free for tests to use.  Note that no subtest should rely on any state changes introduced by a previous subtest.