By Gavin JT Powell (BSc, OCP)
October 8th, 2002
What is an Index?
An index is used to increase read access performance. A book, having an index, allows rapid access to a particular subject area within that book. Indexing a database table provides rapid location of specific rows within that table, where indexes are used to optimize the speed of access to rows. When indexes are not used or are not matched by SQL statements submitted to that database then a full table scan is executed. A full table scan will read all the data in a table to find a specific row or set of rows, this is extremely inefficient when there are many rows in the table.
It is often more efficient to full table scan small tables. The optimizer will often assess full table scan on small tables as being more efficient than reading both index and data space, particularly where a range scan rather than an exact match would be used against the index.
An index of columns on a table contains a one-to-one ratio of rows between index and indexed table, excluding binary key groupings, more on this later. An index is effectively a separate table to that of the data table. Tables and indexes are often referred to as data and index spaces. An index contains the indexed columns plus a ROWID value for each of those column combination rows. When an index is searched through the indexed columns rather than all the data in the row of a table is scanned. The index space ROWID is then used to access the table row directly in the data space. An index row is generally much smaller than a table row, thus more index rows are stored in the same physical space, a block. As a result less of the database is accessed when using indexes as opposed to tables to search for data. This is the reason why indexes enhance performance.
The Basic “How to” of Indexing
There are a number of important factors with respect to efficient and effective creation and use of indexing.
The number of indexes per table.
The number of table columns to be indexed.
What datatypes are sensible for use in columns to be indexed?
Types of indexes from numerous forms of indexes available.
How does SQL behave with indexes?
What should be indexed?
What should not be indexed?
Number of Indexes per Table
Whenever a table is inserted into, updated or deleted from, all indexes plus the table must be updated. Thus if one places ten indexes onto a single table then every change to that table requires an effective change to a single table and ten indexes. The result is that performance will be substantially degraded since one insert requires eleven inserts to insert the new row into both data and index spaces. Be frugal with indexing and be conscious of the potential ill as well as the good effects produced by indexing. The general rule is that the more dynamic a table is the fewer indexes it should have.
A dynamic table is a table changes constantly, such as a transactions table. Catalog tables on the other hand store information such as customer details; customers change a lot less often than invoices. Customer details are thus static in nature and over-indexing may be advantageous to performance.
Number of Columns to Index
Composite indexes are indexes made up of multiple columns. Minimize on the number of columns in a composite key. Create indexes with single columns. Composite indexes are often a requirement of traditional relational database table structures.
With the advent of object-oriented application programming languages such as Java, sequence identifiers tend to be used to identify every row in every table uniquely. The result is single column indexes for every table. The only exceptions are generally many-to-many join resolution entities.
It may sometimes be better to exclude some of the lower-level or less relevant columns from the index since at that level there may not be much data, if there are not many rows to index it can be more efficient to read a group of rows from the data space. For instance, a composite index comprised of five columns could be reduced to the first three columns based on a limited number of rows traversed as a result of ignoring the last two columns. Look at your data carefully when constructing indexes. The more columns you add to a composite index the slower the search will be since there is a more complex requirement for that search and the indexes get physically larger. The benefit of indexes is that an index occupies less physical space than the data. If the index gets so large that it is as large as the data then it will become less efficient to read both the index and data spaces rather than just the data space.
Most database experts recommend a maximum of three columns for composite keys.
Datatypes of Index Columns
Integers make the most efficient indexes. Try to always create indexes on columns with fixed length values. Avoid using VARCHAR2 and any object data types. Use integers if possible or fixed length, short strings. Also try to avaoid indexing on dates and floating-point values. If using dates be sure to use the internal representation or just the date, not the date and the time. Use integer generating sequences wherever possible to create consistently sequential values.
Types of Indexes
There are different types of indexes available in different databases. These different indexes are applicable under specific circumstances, generally for specific search patterns, for instance exact matches or range matches.
The simplest form of indexing is no index at all, a heap structure. A heap structure is effectively a collection of data units, rows, which is completely unordered. The most commonly used indexed structure is a Btree (Binary Tree). A Btree index is best used for exact matches and range searches. Other methods of indexing exist.
1. Hashing algorithms produce a pre-calculated best guess on general row location and are best used for exact matches.
2. ISAM or Indexed Sequential Access Method indexes are not used in Oracle.
3. Bitmaps contain maps of zero’s and 1’s and can be highly efficient access methods for read-only data.
4. There are other types of indexing which involve clustering of data with indexes.
In general every index type other than a Btree involves overflow. When an index is required to overflow it means that the index itself cannot be changed when rows are added, changed or removed. The result is inefficiency because a search to find overflowing data involves a search through originally indexed rows plus overflowing rows. Overflow index space is normally not ordered. A Btree index can be altered by changes to data. The only exception to a Btree index coping with data changes in Oracle is deletion of rows. When rows are deleted from a table, physical space previously used by the index for the deleted row is never reclaimed unless the index is rebuilt. Rebuilding of Btree indexes is far less common than that for other types of indexes since non-Btree indexes simply overflow when row changes are applied to them.
Oracle uses has the following types of indexing available.
Btree index. A Btree is a binary tree. General all-round index and common in OLTP systems. An Oracle Btree index has three layers, the first two are branch node layers and the third, the lowest, contains leaf nodes. The branch nodes contain pointers to the lower level branch or leaf node. Leaf nodes contain index column values plus a ROWID pointer to the table row. The branch and leaf nodes are optimally arranged in the tree such that each branch will contain an equal number of branch or leaf nodes.
Bitmap index. Bitmap containing binary representations for each row. A zero implies that a row does not have a specified value and a 1 denotes that row having that value. Bitmaps are very susceptible to overflow in OLTP systems and should only be used for read-only data such as in Data Warehouses.
Function-Based index. Contains the result of an expression pre-calculated on each row in a table.
Index Organized Tables. Clusters index and data spaces together physically for a single table and orders the merged physical space in the order of the index, usually the primary key. An index organized table is a table as well as an index, the two are merged.
Clusters. Partial merge of index and data spaces, ordered by an index, not necessarily the primary key. A cluster is similar to an index organized table except that it can be built on a join (more than a single table). Clusters can be ordered using binary tree structures or hashing algorithms. A cluster could also be viewed as a table as well as an index since clustering partially merges index and data spaces.
Bitmap Join index. Creates a single bitmap for one table in a join.
Domain index. Specific to certain application types using contextual or spatial data, amongst others.
Indexing Attributes
Various types of indexes can have specific attributes or behaviors applied to them. These behaviors are listed below, some are Oracle specific and some are not.
Ascending or Descending. Indexes can be order in either way.
Uniqueness. Indexes can be unique or non-unique. Primary keys must be unique since a primary key uniquely identifies a row in a table referentially. Other columns such as names sometimes have unique constraints or indexes, or both, added to them.
Composites. A composite index is an index made up of more than one column in a table.
Compression. Applies to Btree indexes where duplicated prefix values are removed. Compression speeds up data retrieval but can slow down table changes.
Reverse keys. Bytes for all columns in the index are reversed, retaining the order of the columns. Reverse keys can help performance in clustered server environments (Oracle8i Parallel Server / RAC Oracle9i) by ensuring that changes to similar key values will be better physically spread. Reverse key indexing can apply to rows inserted into OLTP tables using sequence integer generators, where each number is very close to the previous number. When searching for and updating rows with sequence identifiers, where rows are searched for
Null values. Null values are generally not included in indexes.
Sorting (NOSORT). This option is Oracle specific and does not sort an index. This assumes that data space is physically ordered in the desired manner.
What SQL does with Indexes
In general a SQL statement will attempt to match the structure of itself to an index, the where clause ordering will attempt to match available indexes and use them if possible. If no index is matched then a full table scan will be executed. A table scan is extremely inefficient for anything but the smallest of tables. Obviously if a table is read sequentially, in physical order then an index is not required. A table does not always need an index.
What to Index
Use indexes where frequent queries are performed with where and order by clause matching the ordering of columns in those indexes. Use indexing generally on larger tables or multi-table, complex joins. Indexes are best created in the situations listed below.
Columns used in joins.
Columns used in where clauses.
Columns used in order by clauses.
In most relational databases the order-by clause is generally executed on the subset retrieved by the where clause, not the entire data space. This is not always unfortunately the case for Oracle.
Traditionally the order-by clause should never include the columns contained in the where cause. The only case where the order-by clause will include columns contained in the where clause is the case of the where clause not matching any index in the database or a requirement for the order by clause to override the sort order of the where, typically in highly complex, multi-table joins.
The group-by clause can be enhanced by indexing when the range of values being grouped is small in relation to the number of rows in the table selected.
What not to Index
Indexes will degrade performance of inserts, updates and deletes, sometimes substantially.
Tables with a small number of rows.
Static tables.
Columns with a wide range of values.
Tables changed frequently and with a low amount of data retrieval.
Columns not used in data access query select statements.
Tuning Oracle SQL Code and Using Indexes
What is SQL Tuning?
Tune SQL based on the nature of your application, OLTP or read-only Data Warehouse. OLTP applications have high volumes of concurrent transactions and are better served with exact match SQL where many transactions compete for small amounts of data. Read-only Data Warehouses require rapid access to large amounts of information at once and thus many records are accessed at once, either by many or a small number of sessions.
The EXPLAIN PLAN command can be used to compare different versions of SQL statements, and tune your application SQL code as required. When tuning OLTP applications utilize sharing of SQL code in PL/SQL procedures and do not use triggers unless absolutely necessary. Triggers can cause problems such as self-mutating transactions where a table can expect a lock on a row already locked by the same transaction. This is because triggers do not allow transaction termination commands such as COMMIT and ROLLBACK. In short, do not use triggers unless absolutely necessary.
The best approach to tuning of SQL statements is to seek out those statements consuming the greatest amount of resources (CPU, memory and I/O). The more often a SQL statement is executed the more finely it should be tuned. Additionally SQL statements executed many times more often than other SQL statements can cause issues with locking. SQL code using bind variables will execute much faster than those not. Constant re-parsing of similar SQL code can over stress CPU time resources.
Tuning is not necessarily a never-ending process but can be iterative. It is always best to take small steps and then assess improvements. Small changes are always more manageable and more easier to implement. Use the Oracle performance views plus tools such as TKPROF, tracing, Oracle Enterprise Manager, Spotlight, automated scripts and other tuning tools or packages which aid in monitoring and Oracle performance tuning. Detection of bottlenecks and SQL statements causing problem is as important as resolving those issues. In general tuning falls into three categories as listed below, in order of importance and performance impact.
1. Data model tuning.
2. SQL statement tuning.
3. Physical hardware and Oracle database configuration.
Physical hardware and Oracle database configuration installation will, other than bottleneck resolution, generally only affect performance by between 10% and 20%. Most performance issues occur from poorly developed SQL code, with little attention to SQL tuning during development, probably causing around 80% of general system performance problems. Poor data model design can cause even more serious performance problems than SQL code but it is rare because data models are usually built more carefully than SQL code. It is a common problem that SQL code tuning is often left to DBA personnel. DBA people are often trained as Unix Administrators, SQL tuning is conceptually a programming skill; programming skills of Unix Administrators are generally very low-level, if present at all, and very different in skills requirements to that of SQL code.
How to Tune SQL
Indexing
When building and restructuring of indexing never be afraid of removing unused indexes. The DBA should always be aware of where indexes are used and how.
Oracle9i can automatically monitor index usage using the ALTER INDEX index name [NO]MONITORING USAGE; command with subsequent selection of the USED column from the V$OBJECT_USAGE column.
Taking an already constructed application makes alterations of any kind much more complex. Pay most attention to indexes most often utilized. Some small static tables may not require indexes at all. Small static lookup type tables can be cached but will probably be force table-scanned by the optimizer anyway; table-scans may be adversely affected by the addition of unused superfluous indexes. Sometimes table-scans are faster than anything else. Consider the use of clustering, hashing, bitmaps and even index organized tables, only in Data Warehouses. Many installations use bitmaps in OLTP databases, this often a big mistake! If you have bitmap indexes in your OLTP database and are having performance problems, get rid of them!
Oracle recommends the profligate use of function-based indexes, assuming of course there will not be too many of them. Do not allow too many programmers to create their indexes, especially not function-based indexes, because you could end-up with thousands of indexes. Application developers tend to be unaware of what other developers are doing and create indexes specific to a particular requirement where indexes may be used in only one place. Some DBA control and approval process must be maintained on the creation of new indexes. Remember, every table change requires a simultaneous update to all indexes created based on that table.
SQL Statement Reorganisation
SQL statement reorganization encompasses factors as listed below, amongst others.
WHERE clause filtering and joining orders matching indexes.
Use of hints is not necessarily a good idea. The optimizer is probably smarter than you are.
Simplistic SQL statements and minimizing on table numbers in joins.
Use bind variables to minimize on re-parsing. Buying lots of expensive RAM and sizing your shared pool and database buffer cache to very large values may make performance worse. Firstly, buffer cache reads are not as fast as you might think. Secondly, a large SQL parsing shared pool, when not using bind variables in SQL code, will simply fill up and take longer for every subsequent SQL statement to search.
Oracle9i has adopted various SQL ANSI standards. The ANSI join syntax standard could cause SQL code performance problems. The most effective tuning approach to tuning Oracle SQL code is to remove rows from joins using where clause filtering prior to joining multiple tables, obviously the larger tables, requiring the fewest rows should be filtered first. ANSI join syntax applies joins prior to where clause filtering; this could cause major performance problems.
Nested subquery SQL statements can be effective under certain circumstances. However, nesting of SQL statements increases the level of coding complexity and if sometimes looping cursors can be utilized in PL/SQL procedures, assuming the required SQL is not completely ad-hoc.
Avoid ad-hoc SQL if possible. Any functionality, not necessarily business logic, is always better provided at the application level. Business logic, in the form of referential integrity, is usually best catered for in Oracle using primary and foreign key constraints and explicitly created indexes.
Nested subquery SQL statements can become over complicated and impossible for even the most brilliant coder to tune to peak efficiency. The reason for this complexity could lie in an over-Normalized underlying data model. In general use of subqueries is a very effective approach to SQL code performance tuning. However, the need to utilize intensive, multi-layered subquery SQL code is often a symptom of a poor data model due to requirements for highly complex SQL statement joins.
Some Oracle Tricks
Use [NOT] EXISTS Instead of [NOT] IN
In the example below the second SQL statement utilizes an index in the subquery because of the use of EXISTS in the second query as opposed to IN. IN will build a set first and EXISTS will not. IN will not utilize indexes whereas EXISTS will.
SELECT course_code, name FROM student
WHERE course_code NOT IN
(SELECT course_code FROM maths_dept);
SELECT course_code, name FROM student
WHERE NOT EXISTS
(SELECT course_code FROM maths_dept
WHERE maths_dept.course_code = student.course_code);
In the example below the nesting of the two queries could be reversed depending on which table has more rows. Also if the index is not used or not available, reversal of the subquery is required if tableB has significantly more rows than tableA.
DELETE FROM tableA WHERE NOT EXISTS
(SELECT columnB FROM tableB WHERE tableB.columnB = tableA.columnA);
Use of value lists with the IN clause could indicate a missing entity. Also that missing entity is probably static in nature and can potentially be cached, although caching causing increased data buffer size requirements is not necessarily a sensible solution.
SELECT country FROM countries WHERE continent IN ('africa','europe','north america');
Equijoins and Column Value Transformations
AND and = predicates are the most efficient. Avoid transforming of column values in any form, anywhere in a SQL statement, for instance as shown below.
SELECT * FROM
No comments:
Post a Comment