Using Memory Maps to Speed Up Loading of Large GDSII files

Memory Maps

In order to display or otherwise work with a GDSII file, QISLIB must first scan it to determine the contents -- cells, references, entities and so on. It then makes a second pass and loads into memory a quad tree which divides the layout into regions and contains pointers to entity data in the actual file. The entities may remain on disk or be loaded into RAM (depending on whether there is enough RAM to hold the entities.) This scanning of the file and building the quad tree and associated pointers can take a significant amount of time for large files.

If one opens the same file over and over again (in different sessions) the scan/load time becomes a significant overhead. The overhead can be eliminated by producing a memory map -- essentially a copy of what QISLIB places into memory upon scanning/loading a file.

Once the memory map has been produced (and saved to disk) the next time the user wishes to display or otherwise process the GDSII layout, the memory map is opened. The time required is really only limited by the disk IO read time.

OASIS Limitation

Memory maps can only be produced for GDSII stream files -- not for OASIS files. The reasons for this are complex and related to how we organize OASIS data vs GDSII data. So the following instructions apply solely to GDSII input.

Generating the Memory Map

When handling large GDSII files it is common to pass in only the layers you need to deal with. We're making that assumption here. Here is the order in which you would call the QISLIB funcitons to open a file called demo1.gds and produce a memory map in the directory c:\tmp.


QisLib_SetMemoryMapsDir("c:\tmp");                   // define directory where the
                                                     // memory map will be written.
													 
QisLib_SetCreateMemoryMaps(_QIS_ON);                 // Turns on the memory map creation flag.

QisLib_SetLoadMemory(_QIS_ON);                       // loads the entity database into RAM;
                                                     // it requires more RAM but is a
                                                     // requirement for independent memory map.
      
QisLib_SetInputLayerMap("All-NULL,7:3-7:3,7:4-7:4"); // input layer filter; blocks all, then
                                                     // passes layer:datatype 7:3 and 7:4
													 
QisLib_OpenGDSII("demo1.gds");                       // open the GDSII file

This set of calls will generate a pair of memory maps: demo1.gds.scan.l64 and demo1.gds.dbload.l64 in the c:\tmp directory. The dbload means the memory maps has ALL the entity information of the GDSII file on layers 7:3 and 7:4 but no data that belonged to any other layer of the source file.

This map file pair is now independent of the original file that produced them. You could delete the source file and still use QISLIB to open and view the layout -- but only layers 7:3 and 7:4 as we blocked all the other entity data when we created memory map.

Loading the Memory Map

Now that you have created a memory map pair, it is very fast to load. You have two choices when loading this pair. If SetLoadMemory=ON then the dbload file will be copied into memory. This takes a bit more time initially but subsequent operations will be faster. If SetLoadMemory=OFF then the dbload map file is left on disk. This starts up faster but each operation such as a Get_Vector will require disk IO.


QisLib_SetLoadMemory(_QIS_ON);                                        // uses more memory, slower open, 
                                                                      // but faster operation after 
                                                                      it's opened


// *** OR ***


QisLib_SetLoadMemory(_QIS_OFF);                                       // uses less memory, faster open, 
                                                                      //slower operation after it's opened




QisLib_OpenMemoryMaps("demo1.gds.scan.l64","demo1.gds.dbload.l64");   //instead of OpenGDSII function use
                                                                      //and point to the map files.

Why Use Memory Maps?

We originally developed the memory map for users who worked in a reliability lab with equipment such as a FIB (focused ion beam machine.) The client of the lab rents the FIB at a very high rate -- anywhere from $500 to $1000 per hour. The layout data needs to be viewed on the FIB's workstation in order to help the user navigate about and probe the chip. Loading a large GDSII layout file that takes 15 minutes gets very costly. So an offline computer (not connected to the FIB) is used to create the memory map files which load onto the FIB computer in less than a minute.

[The offline computer must use the same byte order as the destination computer -- so you can't create a memory map on Windows and then use it on a Linux computer as these two different OS's don't share the same byte order.]