Page 1 of 1

VASP on Altix

Posted: Thu Aug 25, 2005 3:21 pm
by tjf
I have VASP 4.6 running happily on an SGI Altix. (The details of the machine are on the SARA website for those that care, http://www.sara.nl/userinfo/aster/descr ... index.html)

As I've read here somewhere, ifort 8.1, parallel VASP and FFTW don't play nicely together. The VASP built in FFTs work a charm, though.

I based my makefile on the efc itanium makefile.

One issue which isn't really clear to me though is how to set CACHE_SIZE. Some timing tests show the results below, where the CACHE_SIZE on the left gives a runtime on the right, in seconds.

0 10259
1000 6627
3000 4845
5000 4709
8000 5406
10000 4701
15000 4335
20000 8454
30000 6883
50000 5162
100000 6951

These are single runs on 8 CPUs, so other stuff is clearly going on to screw up the timings. Still, I'm using 15000. What would one expect to be optimal on a machine with 3 MB cache per processor? (I do assume that the CACHE_SIZE parameter should be set on a per processor basis, not on cache over the whole CPU set.)

Anyone else care to share their experiences with Altix/other Itanium SMP?

VASP on Altix

Posted: Mon Oct 24, 2005 2:28 pm
by fish
Could you post your makefile?

VASP on Altix

Posted: Wed Nov 02, 2005 4:19 pm
by hilgeman
Hi,

You should set the CACHE_SIZE to the L2 size of the Itanium2 processor, which is 256 kB. That equivalents in a CACHE_SIZE value of 16000.

Also, I recommend you to either use the built-in FFT functions from Jurgen Furthmuller or the FFT functions from SGI's SCSL scientific library (compile with -DFFT_SCSL to use that).

I (SGI) have a patch on top of VASP 4.6.26/27/28 that has been sent to Georg Kresse,which has among other preformace enhancements, an alternative shared memory implementation of the collective MPI routine to distribute the in-band wavefunctions. The Altix is a *very* fast machine for running VASP, please send me a private email when you would like to have the patch.

regards,

-Martin

--
Martin Hilgeman Chemical Applications Engineering
Phone: +31(0)30-6696885
SGI E-mail: hilgeman@sgi.com
The Netherlands URL: http://www.sgi.com/go/chembio

VASP on Altix

Posted: Fri Nov 04, 2005 2:21 pm
by tjf
My abbreviated but functional makefile is below.

It's a bit of a hack. YMMV.

Code: Select all

.SUFFIXES: .inc .f .f90 .F
# all CPP processed fortran files have the extension .f
SUFFIX=.f90
FC=ifort
FCL=$(FC)
CPP_ =  ./preprocess <$*.F | /usr/bin/cpp -P -C -traditional >$*$(SUFFIX)
FFLAGS =  -FR -lowercase -cm -w95 -tpp2  -safe_cray_ptr -stack_temps \
         -I/usr/local/opt/fftw-3.0.1-fma/include
OFLAG=-O3 -unroll0 -ivdep_parallel -fno-alias -ip
OFLAG_HIGH = $(OFLAG)
OBJ_HIGH =
OBJ_NOOPT =
DEBUG  = -O0
INLINE = $(OFLAG)
BLAS=-L/usr/local/opt/scs_beta/lib/ -lscs
LINK    =
FFT3D   = fft3dfurth.o fft3dlib.o
CPP    = $(CPP_) -DMPI  -DHOST=\"LinuxAltix\" -DIFC -Duse_collective \
     -Dkind8 -DNGZhalf -DCACHE_SIZE=16000 -DPGF90 -Davoidalloc \
     -DMPI_BLOCK=500 -DPROC_GROUP=8
LIB     = -L../vasp.4.lib -ldmy  \
      ../vasp.4.lib/linpack_double.o \
      -L/usr/local/opt/scs_beta/lib/ -lscs \
      -lmpi
FFT3D   = fftmpi.o fftmpi_map.o fft3dlib.o
                                                                                                                              
BASIC=   symmetry.o symlib.o   lattlib.o  random.o
                                                                                                                              
SOURCE=  base.o     mpi.o      smart_allocate.o      xml.o  \
         constant.o jacobi.o   main_mpi.o  scala.o   \
         asa.o      lattice.o  poscar.o   ini.o      setex.o     radial.o  \
         pseudo.o   mgrid.o    mkpoints.o wave.o      wave_mpi.o  $(BASIC) \
         nonl.o     nonlr.o    dfast.o    choleski2.o    \
         mix.o      charge.o   xcgrad.o   xcspin.o    potex1.o   potex2.o  \
         metagga.o  constrmag.o pot.o      cl_shift.o force.o    dos.o      elf.o      \
         tet.o      hamil.o    steep.o    \
         chain.o    dyna.o     relativistic.o LDApU.o sphpro.o  paw.o   us.o \
         ebs.o      wavpre.o   wavpre_noio.o broyden.o \
         dynbr.o    rmm-diis.o reader.o   writer.o   tutor.o xml_writer.o \
         brent.o    stufak.o   fileio.o   opergrid.o stepver.o  \
         dipol.o    xclib.o    chgloc.o   subrot.o   optreal.o   davidson.o \
         edtest.o   electron.o shm.o      pardens.o  paircorrection.o \
         optics.o   constr_cell_relax.o   stm.o    finite_diff.o \
         elpol.o    setlocalpp.o
                                                                                                                              
INC=
                                                                                                                              
vasp: $(SOURCE) $(FFT3D) $(INC) main.o
        rm -f vasp
        $(FCL) -o vasp $(LINK) main.o  $(SOURCE)   $(FFT3D) $(LIB)
makeparam: $(SOURCE) $(FFT3D) makeparam.o main.F $(INC)
        $(FCL) -o makeparam  $(LINK) makeparam.o $(SOURCE) $(FFT3D) $(LIB)
zgemmtest: zgemmtest.o base.o random.o $(INC)
        $(FCL) -o zgemmtest $(LINK) zgemmtest.o random.o base.o $(LIB)
dgemmtest: dgemmtest.o base.o random.o $(INC)
        $(FCL) -o dgemmtest $(LINK) dgemmtest.o random.o base.o $(LIB)
ffttest: base.o smart_allocate.o mpi.o mgrid.o random.o ffttest.o $(FFT3D) $(INC)
        $(FCL) -o ffttest $(LINK) ffttest.o mpi.o mgrid.o random.o smart_allocate.o base.o $(FFT3D) $(LIB)
kpoints: $(SOURCE) $(FFT3D) makekpoints.o main.F $(INC)
        $(FCL) -o kpoints $(LINK) makekpoints.o $(SOURCE) $(FFT3D) $(LIB)
                                                                                                                              
clean:
        -rm -f *.f *.o *.L ; touch *.F
                                                                                                                              
main.o: main$(SUFFIX)
        $(FC) $(FFLAGS)$(DEBUG)  $(INCS) -c main$(SUFFIX)
xcgrad.o: xcgrad$(SUFFIX)
        $(FC) $(FFLAGS) $(INLINE)  $(INCS) -c xcgrad$(SUFFIX)
xcspin.o: xcspin$(SUFFIX)
        $(FC) $(FFLAGS) $(INLINE)  $(INCS) -c xcspin$(SUFFIX)
                                                                                                                              
makeparam.o: makeparam$(SUFFIX)
        $(FC) $(FFLAGS)$(DEBUG)  $(INCS) -c makeparam$(SUFFIX)
                                                                                                                              
makeparam$(SUFFIX): makeparam.F main.F
base.o: base.inc base.F
mgrid.o: mgrid.inc mgrid.F
constant.o: constant.inc constant.F
lattice.o: lattice.inc lattice.F
setex.o: setexm.inc setex.F
pseudo.o: pseudo.inc pseudo.F
poscar.o: poscar.inc poscar.F
mkpoints.o: mkpoints.inc mkpoints.F
wave.o: wave.inc wave.F
nonl.o: nonl.inc nonl.F
nonlr.o: nonlr.inc nonlr.F
                                                                                                                              
$(OBJ_HIGH):
        $(CPP)
        $(FC) $(FFLAGS) $(OFLAG_HIGH) $(INCS) -c $*$(SUFFIX)
$(OBJ_NOOPT):
        $(CPP)
        $(FC) $(FFLAGS) $(INCS) -c $*$(SUFFIX)
                                                                                                                              
fft3dlib_f77.o: fft3dlib_f77.F
        $(CPP)
        $(F77) $(FFLAGS_F77) -c $*$(SUFFIX)
fft3dfurth.o : fft3dfurth.F
        $(CPP)
        $(FC) $(FFLAGS) -O2 -c $*$(SUFFIX)
                                                                                                                              
fftmpi.o : fftmpi.F
        $(CPP)
        $(FC) $(FFLAGS) -O2 -c $*$(SUFFIX)
                                                                                                                              
symlib.o : symlib.F
        $(CPP)
        $(FC) $(FFLAGS) -O1 -c $*$(SUFFIX)
                                                                                                                              
symmetry.o : symmetry.F
        $(CPP)
        $(FC) $(FFLAGS) -O1 -c $*$(SUFFIX)
                                                                                                                              
dynbr.o : dynbr.F
        $(CPP)
        $(FC) $(FFLAGS) -O1 -c $*$(SUFFIX)
                                                                                                                              
broyden.o : broyden.F
        $(CPP)
        $(FC) $(FFLAGS) -O1 -c $*$(SUFFIX)
                                                                                                                              
us.o : us.F
        $(CPP)
        $(FC) $(FFLAGS) -O1 -c $*$(SUFFIX)
                                                                                                                              
wave.o : wave.F
        $(CPP)
        $(FC) $(FFLAGS) -O0 -c $*$(SUFFIX)

LDApU.o : LDApU.F
        $(CPP)
        $(FC) $(FFLAGS) -O2 -c $*$(SUFFIX)
                                                                                                                              
potex1.o : potex1.F
        $(CPP)
        $(FC) $(FFLAGS) -O2 -c $*$(SUFFIX)
                                                                                                                              
.F.o:
        $(CPP)
        $(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)
.F$(SUFFIX):
        $(CPP)
$(SUFFIX).o:
        $(FC) $(FFLAGS) $(OFLAG) $(INCS) -c $*$(SUFFIX)


<span class='smallblacktext'>[ Edited Fri Nov 04 2005, 03:30PM ]</span>

VASP on Altix

Posted: Fri Nov 04, 2005 5:36 pm
by tjf
Well, it was functional before the forum software butchered it.

The important stuff is at the top. I also reduced the optimisation for potex1 and I think I added those generic rules at the bottom, but I may have picked them up from somewhere else. They don't seem to be in the efc makefile.