Thursday, September 25, 2014

Join Index

Join Index

A Join Index is an optional index which may be created by a User. Join indexes provide additional processing efficiencies:
  • Eliminate base table access
  • Eliminate aggregate processing
  • Reduce joins
  • Eliminate redistributions
  • Eliminate Summary processing
The following are wide variety of Join Index names. We will discuss each in clear detail.
  1. Single Table Join Index
  2. Multi-Table Join Index
  3. Multi-Table Compressed Join Index
  4. Aggregate Join Index
  5. Sparse Join Index
  6. Global Join Index
  7. Hash Index
A Single-Table Join Index duplicates a single table, but changes the Primary Index. Users will only query the base table, but the Parsing Engine will use the Join Index.
TeradataWiki-Teradata Single-Table Join Index

The reason to create a Single-Table Join Index is so joins can be performed faster because no Redistributions or Duplication needs to occur.

A Multi-Table Join index is a Join Index that involves two or more tables.
Facilitates join operations by possibly eliminating join processing or by reducing/eliminating join data redistribution.
TeradataWiki-Teradata Multi-Table Join index

A Compressed Join Index is designed to save space by not REPEATING the repeating values
TeradataWiki-Teradata Compressed Join Index

An Aggregate Join Index will allow tracking of the Aggregates SUM and COUNT on any table
TeradataWiki-Teradata Aggregate Join Index

A Sparse Join Index is a Join Index that doesn‘t use every row because it has a WHERE Clause. This is done to save space and time.
TeradataWiki-Teradata Sparse Join Index
Hash Indexes are used similar to a Join Index, but Hash Indexes are maintained in AMP-Local tables and used to quickly find certain key columns in a base table.
TeradataWiki-Teradata Hash Indexes
Join Index Details:
  • Max 64 columns per table per Join Index.
  • BLOB and CLOB types cannot be defined.
  • Triggers with Join Indexes are allowed V2R6.
  • After Restoring a Table, Drop and Recreate the Join Index.
  • Automatically updated as table changes.
  • Fast load/Multi load wont load with them.
  • can have NUPI and NUSI.
  • Collect statistics on Primary and Secondary.

Secondary Index

Secondary Index

Before going to learn about Secondary Index, i would like to suggest you learn Primary Index.

A Secondary Index (SI) is an alternate data access path. It allows you to access the data without having to do a full-table scan.

You can drop and recreate secondary indexes dynamically, as they are needed.Secondary Indexes are stored in separate subtables that requires additional disk space and maintenance which is handled automatically by the system.

The entire purpose for the Secondary Index Subtable will be to point back to the real row in the base table via the Row-ID. 

Secondary Index Rules
Rule 1: Secondary Indexes are optional. 
Rule 2: Secondary Index values can be unique or non-unique.
Rule 3: Secondary Index values can be NULL.
Rule 4: Secondary Index values can be modified.
Rule 5: Secondary Indexes can be changed.
Rule 6: A Secondary Index has a limit of 64 columns.

Like Primary Indexes,Secondary Indexes also two Types (USI or NUSI).

A Unique Secondary Index (USI) serves two purposes. 
  • Enforces uniqueness on a column or group of columns. The database will check USIs to see if the values are unique.
  • Speeds up access to a row (data retrieval speed).
When A USI is created Teradata will immediately build a secondary index subtable on each AMP.
Each AMP will then hash the secondary index value for each of their rows in the base table.

In the Below diagram,each AMP hashes the Emp_no column for all employee rows they hold.

The output of the Emp_no hash will utilize the hash map to point to a specific AMP and that AMP will hold the secondary index subtable row for the secondary index value. 

That means the subtable row will hold the base table Row-ID and Teradata will then find the base row immediately.

Syntax to create a Unique Secondary Index is:
CREATE UNIQUE INDEX (Column/Columns) ON .;
TeradataWiki-Teradata USI

A Non-Unique Secondary Index (NUSI) is usually specified to prevent full-table scans, in which every row of a table is read. 

When A NUSI is created Teradata will immediately build a secondary index subtable on each AMP.

Each AMP will hold the secondary index values for their rows in the base table only. In our example, each AMP holds the name column for all employee rows in the base table on their AMP (AMP local).

Each AMP Local name will have the Base Table Row-ID (pointer)so the AMP can retrieve it quickly if needed. If an AMP contains duplicate first names, only one subtable row for that name is built with multiple Base Row-IDs.

Syntax to create a Non-Unique Secondary Index is:
CREATE INDEX (Column/Columns) ON .;

There can be up to 32 Secondary Indexes on a table
An USI is always a Two-AMP operation so it is almost as fast as a Primary Index, but a NUSI is an All-AMP operation, but not a Full Table Scan.


Data Access With USI:

TeradataWiki-Teradata USI
  • The SQL is submitted, specifying a USI (in this case, a customer number of 56).
  • The hashing algorithm calculates a row hash value (in this case, 602).
  • The hash map points to the AMP containing the subtable row corresponding to the row hash value (in this case, AMP 2).
  • The subtable indicates where the base row resides (in this case, row 778 on AMP 4).
  • The message goes back over the BYNET to the AMP with the row and the AMP accesses the data row (in this case, AMP 4).
  • The row is sent over the BYNET to the PE, and the PE sends the answer set on to the client application.
  • As shown in the example above, accessing data with a USI is typically a two-AMP operation. However, it is possible that the subtable row and base table row could end up being stored on the same AMP, because both are hashed separately. If both were on the same AMP, the USI request would be a one-AMP operation.

Data Access With NUSI:
TeradataWiki-Teradata USI
  • The SQL is submitted, specifying a NUSI (in this case, a last name of "Smith").
  • The hashing algorithm calculates a row hash value for the NUSI (in this case, 567).
  • All AMPs are activated to find the hash value of the NUSI in their index subtables. The AMPs whose subtables contain that value become the participating AMPs in this request (in this case, AMP1 and AMP2). The other AMPs discard the message.
  • Each participating AMP locates the row IDs (row hash value plus uniqueness value) of the base rows corresponding to the hash value (in this case, the base rows corresponding to hash value 567 are 640, 222, and 115).
  • The participating AMPs access the base table rows, which are located on the same AMP as the NUSI subtable (in this case, one row from AMP 1 and two rows from AMP 2).
  • The qualifying rows are sent over the BYNET to the PE, and the PE sends the answer set on to the client application (in this case, three qualifying rows are returned).

Primary Index

Primary Index


Every table must have at least one column as the Primary Index. The Primary Index is defined when the table is created.There are two reasons you might pick a different Primary Index then your Primary Key. They are (1) for Performance reasons and (2) known access paths.

Primary Index Rules
Rule 1: One Primary Index per table.
Rule 2: A Primary Index value can be unique or non-unique.
Rule 3: The Primary Index value can be NULL.
Rule 4: The Primary Index value can be modified.
Rule 5: The Primary Index of a populated table cannot be modified.
Rule 6: A Primary Index has a limit of 64 columns.

Two Types of Primary Indexes (UPI or NUPI)

Unique Primary Index(UPI)
A Unique Primary Index (UPI) is unique and cannot have any duplicates.
If you try and insert a row with a Primary Index value that is already in the table, the row will be rejected. An UPI enforces UNIQUENESS for a column.
 
A Unique Primary Index (UPI) will always spread the rows of the table evenly amongst the AMPs. UPI access is always a one-AMP operation. 
 
We have selected EMP_NO to be our Primary Index. Because we have designated EMP_NO to be a Unique Primary Index, there can be no duplicate employee numbers in the table.         
TeradataWiki-Teradata UPI

Non-Unique Primary Index (NUPI) 
A Non-Unique Primary Index (NUPI) means that the values for the selected column can be non-unique.Duplicate values can exist.

A Non-Unique Primary Index will almost never spread the table rows evenly.
An All-AMP operation will take longer if the data is unevenly distributed. You might pick a NUPI over an UPI because the NUPI column may be more effective for query access and joins.

We have selected LAST_NAME to be our Primary Index. Because we have designated LAST_NAME to be a Non-Unique Primary Index we are anticipating that there will be individuals in the table with the same last name.
TeradataWiki-Teradata UPI

Multi-Column Primary Indexes: 
Teradata allows more than one column to be designated as the Primary Index. It is still only one Primary Index, but it is merely made up by combining multiple columns together. Teradata allows up to 64 combined columns to make up the one Primary Index required for a table.

On the following page you can see we have designated First_Name and Last_Name combined to make up the Primary Index.
 This is often done for two reasons:
 (1) To get better data distribution among the AMPs
 (2) Users often use multiple keys consistently to query
           
SQL Syntax for Creating a Primary Index

 Creating a Unique Primary Index  
The SQL syntax to create a Unique Primary Index is:
CREATE TABLE sample_1
(col_a INT
,col_b INT
,col_c INT)
UNIQUE PRIMARY INDEX (col_b);

Creating a Non-Unique Primary Index 
The SQL syntax to create a Non-Unique Primary Index is:

CREATE TABLE sample_2
(col_x INT
,col_y INT
,col_z INT)
PRIMARY INDEX (col_x);


Data distribution using Primary Index 
When a user submits an SQL request against a table using a Primary Index, the request becomes a one-AMP operation, which is the most direct and efficient way for the system to find a row. The process is explained below.

Teradata Primary Index

Hashing Process  
1.The primary index value goes into the hashing algorithm.
2.The output of the hashing algorithm is the row hash value.
3.The hash map points to the specific AMP where the row resides.
4.The PE sends the request directly to the identified AMP.
5.The AMP locates the row(s) on its vdisk.
6.The data is sent over the BYNET to the PE, and the PE sends the answer set on to the client application.


Duplicate Row Hash Values 

It is possible for the hashing algorithm to end up with the same row hash value for two different rows.

There are two ways this could happen:

  • Duplicate NUPI values: If a Non-Unique Primary Index is used, duplicate NUPI values will produce the same row hash value. 
  • Hash synonym: Also called a hash collision, this occurs when the hashing algorithm calculates an identical row hash value for two different Primary Index values. H

To differentiate each row in a table, every row is assigned a unique Row ID. The Row ID is the combination of the row hash value and a uniqueness value.


Row ID = Row Hash Value + Uniqueness Value 

The uniqueness value is used to differentiate between rows whose Primary Index values generate identical row hash values. In most cases, only the row hash value portion of the Row ID is needed to locate the row.
TeradataWiki-Teradata Row Hash

When each row is inserted, the AMP adds the row ID, stored as a prefix of the row.
The first row inserted with a particular row hash value is assigned a uniqueness value of 1. The uniqueness value is incremented by 1 for any additional rows inserted with the same row hash value.

Below Diagrams will show how data resides in AMP.

Even Distribution with an UPI 
TeradataWiki-Teradata


Uneven Distribution with a NUPI 
TeradataWiki-Teradata

Teradata Architecture

Architecture

Teradata relies on three architectural components.
TeradataWiki-Teradata Architecture
Teradata Architecture
Parsing Engine (PE)
The Parsing Engine (PE) is a component that interprets SQL requests, receives input records, and passes data. It sends the messages through the BYNET to the AMPs.
PE is made of following component .
1.Session Control: Major function of Session Control is Logon and Logoff
2.Parsing : The parser interprets sql statements and checks it for proper syntax and evaluate them symmetrically.
3. Optimizing :Developing the least expensive plan to return requested response set.
4. Dispatching : Pass the steps on to the AMP via BYNET.

BYNETs
The BYNET act as message-passing layer. It decides which AMP should receive a message.
Mechanism to communicate.
1. Broadcast 
2. Point to Point

Access Module Processor (AMP)
The AMP is a virtual processor (vproc) designed to managing a portion of the entire database.
It performs all database management functions such as sorting, aggregating, and formatting data.
The AMP receives data from the PE, formats rows, and distributes them to the disk storage units it controls. The AMP also retrieves the rows requested by the Parsing Engine.

Disks
Disks are disk drives associated with an AMP that store the data rows.

Performance Guide 1

Performance Tuning

A) Explain the EXPLAIN: Check for EXPLAIN plan to see how exactly Teradata will be executing the query. Try to understand basic keywords in Explain Plan like confidence level, join strategy used, re-distribution happening or not.

B) Collect STATS: The stats of the columns used join conditions should updated. Secondary Indexes without proper STATS can be of little or no help. Check for STATS status of the table.

C) Use Proper PI: If the Primary index is not properly defined in any one or all of the tables in the query. Check if the PI of target table loaded is unique.

D) Use PPI: If there is Partition Primary Index created on a table, try to use it. If you are not using it in filter condition, it will degrade the performance.

E) No FUNCTIONS in Conditions: Try to avoid using function in join conditions. Ex Applying COALESCE or TRIM etc causes high CPU consumption.

F) Use PPI: If Partition Primary Index is defined in tables try to use it. If you are not using it in filter condition, it will degrade the performance.

G) Same column DATA TYPES:
 Define same data type for the joining columns.

H) Avoid IN clause in filter conditions: When there can be huge number of values in where conditions, better option can be to insert such values in a volatile table and use volatile table with INNER JOIN in the main query.

I) Use Same PI in Source & Target: PI columns also can help in saving the data into disk .If the Source and Target have the same PI, data dump can happen very efficiently form source to target.

J) Collect STATS on VOLATILE table: Collect stats on volatile tables where required can save AMPCPU. Remove stats if already present where it is not getting used.
If the volatile table contains UNIQUE PI, then go for sample stats rather than full stats.

K) DROPPING volatile tables explicitly: Once volatile tables is no more required you can drop those. Donít wait for complete procedure to be over. This will free some spool space immediately and could prove to be very helpful in avoiding No More Spool Space error.

L) NO LOG for volatile tables: Create volatile tables with NO LOG option.

M) Check DBQL Stats: Keep your performance stats accessible. Target the most AMPCPU consuming query first.

N) UPDATE clause: Do not write UPDATE clause with just SET condition and no WHERE condition. Even if the Target/Source has just one row, add WHERE clause for PI column.

O) DELETE & INSERT: Sometimes replacing UPDATE with DELETE & INSERT can save good number of AMPCPU. Check if this holds good for your query.

P) Query SPLITS: Split queries into several smaller queries logically and use volatile tables with proper PI.

Q) Try MSR: If same target table is loaded multiple times, try MSR for several sections. This will speed the final MERGE step into target table and you may see good CPU gain.

R) Try OLAP Functions: 
Check if replacing co-related sub query with OLAP function may result in AMPCPU saving.

S) Avoid DUPLICATE data: If the join columns in the tables involved in the query have duplicates. Use Distinct or Group by, load into a volatile table, collect stats and use the volatile table.

T) Use Proper JOINS: If joins used, donít use right outer, left or full joins where inner joins is sufficient.

U) User proper ALIAS: Check the aliases in the joins. Small mistake could lead to a product join.

V) Avoid CAST: Avoid unnecessary casting for DATE columns. Once defined as DATE, you can compare date columns against each other even when they are in different format. Internally, DATE is stored as INTEGER. CAST is required mainly when you have to compare VARCHAR value as DATE.

W) Avoid UDF: 
Most of the functions are available in Teradata for data manipulations. So avoid User Defined Functions

X) Avoid FULL TABLE SCAN: Try to avoid FTS scenarios like SI should be defined on the columns which are used as part of joins or Alternate access path. Collect stats on SI columns else there are chances where optimizer might go for FTS even when SI is defined on that particular column

Y) Avoid using IN/NOT IN: For large list of values, avoid using IN /NOT IN in SQLs. Write large list values to a temporary table and use this table in the query

Z) Use CONSTANTS: 
Use constants to specify index column contents whenever possible, instead of specifying the constant once, and joining the tables. This may provide a small savings on performance.

Monday, September 1, 2014

PDE not accessible: Cannot open PDE device

 This worked for me.
/etc/init.d/tpa stop
rm /var/opt/teradata/tdtemp/PanicLoopDetected <= said to rm this file in /var/log/messages

/etc/init.d/tpa start

Named Pipe : Teradata


Step by Step : Arc /restore by Named pipe

Fig: Arcmain Data Transfer using the Named-Pipe Access Module

Step 1 : Create Named Pipe by using below linux command.
Mknod ARCPIPE p
Step2 : Create archive file and specify file = Pipe name
.logon TDDEV/arc_user2,user2arc1;
ARCHIVE DATA TABLES
(sichake.tbl1),
release lock, file=ARCPIPE;
LOGOFF;
Step 3: Create restore/copy file and specify file = Pipe name
.logon TDTEST/arc_user2,user2arc1;
COPY DATA TABLES
(sichake.tbl1),
release lock,
file=ARCPIPE;
LOGOFF;
Step 4 : Execute below arcmain commands to archive and copy/restore the database/table
nohup arcmain vb3 sessions=10 checkpoint=0 outlog=swdout errlog=swerr < resttest1.arc &
nohup arcmain vb3 sessions=10 checkpoint=0 outlog=goldout errlog=golderr < arctest1.arc &


Two possible archive job strategies
• All-amp - Database objects are identified at a database or table level. Data from all database amps are sent together.
• Serial Cluster - Database objects are identified to the finer granu-larity of a cluster or a group of clusters. Following a dictionary ar-chive, the clusters are archived and restored/copied one after an-other. This is because in almost all cases the target system will be a different configuration from the source system, and each opera-tion will apply an all-amp exclusive lock on its target objects. In the restore/copy script the "CLUSTER=" option will have to be omit-ted, and the "NO BUILD" option will have to be used. Only after all of the archives have been restored, should a separate "BUILD DATA TABLES" command be executed.








Command Line Options
Options to be added to the arcmain command line. There are others, and documentation can be found in the ARC Reference Manual for them. Additional information on setting the values for these options can be found in later sections of this document.
• sessions=n - The number of data sessions. Each session is a sepa-rate connection to the database, the default is 4 ses-sions, but a starting point for analysis should be 1 ses-sion per amp. Session count is discussed further in the Performance Analysis section.
• checkpoint=0 - suppress checkpointing by arcmain
• outlog=jobout - echo arc output to this file (max 8 characters)
• errlog=joberr - echo arc errors to this file (max 8 characters)
• verbose3 - show detailed progress in arcmain output (optional)
• syntax - parse only, do not execute (optional)
• default=filename - allows always-used options to be kept in a file

When using the Named Pipes Access Module there are some additional parameters that must be included to support the access module:
• iomodule=np_AXSMOD.dll - W2K Named Pipes Access Module
• iomodule=np_axsmod.so - MP-RAS Named Pipes Access Module
• ioparms= - parameters passed to the iomodule at launch
• ioparms=fallback_directory=c:\work log_level=4 - Named Pipe parameters
• filedef=(ARCPIPE,\\.\pipe\ARCPIPE) - W2K, local pipe
• filedef=(ARCPIPE,\\.\targetsys\ARCPIPE) - W2K, remote pipe