How one of America’s largest public power utilities implemented an all-encompassing integrated calibration solution
Salt River Project (SRP), Arizona
About Salt River Project
For more than a century, Salt River Project (SRP) has produced power and delivered water to meet the needs of its customers in the greater Phoenix metropolitan area. Today, as one of the nation’s largest public power utilities, SRP provides reliable electricity and water to more than 1 million customers and employs around 4,500 people. In order to produce power, including generation, transmission and distribution, to this 2,900 square mile service area, SRP uses a combination of hydro, gas turbine, coal fired, nuclear, solar, geothermal and wind generation. Furthermore, SRP is the largest water supplier in the Phoenix metropolitan area, responsible for water transmission and distribution for a 375 square mile service area while managing a 13,000 square mile watershed. This water production, transmission and distribution includes an extensive system of reservoirs, wells, canals and irrigation laterals.
TODAY, AS ONE OF THE NATION’S LARGEST PUBLIC POWER UTILITIES, SRP PROVIDES RELIABLE ELECTRICITY AND WATER TO MORE THAN 1 MILLION CUSTOMERS AND EMPLOYS AROUND 4,500 PEOPLE.
SRP’s calibration history
Jody Damron, a Business Analyst at SRP’s corporate headquarters in Tempe, Arizona, has been serving the company for more than 36 years. In 1986, he first began working as instrument calibration technician at the Navajo Generating Station (NGS), a 3-unit, 2400 megawatt, super critical generating station that went live with the first unit in 1974. The calibration documentation process consisted of a hard copy, index card, paper process. At the time, technicians would take paper cards out into the field and record their findings when they returned. They soon encountered several issues, including lost data, no data security, no reporting and difficulty creating an audit trail. Jody recalls when a state auditor visited the plant and the cards were misplaced. Jody was responsible for explaining to the state inspector why they didn’t have cards readily available. As Jody explains, “it’s a very discomforting place to be when you’re right in the middle of an audit.” However, at the time, NGS did not have the technology to help with this process.
By the early 1990s, Microsoft Excel had been released and Jody was the lead to get all of the information from their calibration cards into an Excel file. This was a step in the right direction, as it provided some data security. By the late 1990s, a new distributed control system (DCS) was deployed at NGS and plant personnel were able to document all of the instrument data for the control system, which was data that never existed before. Later on, NGS decided to improve the database by converting to a Microsoft Access database. The Access database not only included calibration information, but all DCS instrument information as well. As one can imagine, this was a big undertaking and it was the first time IT had to be involved in the process. The IT team took the time to learn about the business objectives and processes, which was a vital step. Jody also gained insight to how IT operated, interpreted information and viewed their objectives. Moreover, this effort resulted in Jody and the IT business analyst writing an ISA technical report titled, TR77.70.01 Tracking and Reporting of Instrument and Control Data. Overall, this phase brought more notable improvements to the process, including secure data and calibration reporting, which allowed easier compliance to state audit reports, links to drawings, and documented DCS points. But it was still an inefficient and difficult process that was costly and required continuous oversight.
Small steps for a big evolution
In 2005, Jody was a member of an International Society of Automation (ISA) Executive Committee. While attending an ISA conference, Jody was introduced to Beamex CMX calibration management software during a technical “how to” presentation. He recognized the importance of the increased functionality that calibration software offered over and beyond the current set up. By this time, Jody had become the instrument supervisor. He decided to take the information back to the site and ask his technicians to review it. He wanted them to help him make such an important decision. They too saw the value, and especially liked the user-friendliness along with the ability to capture more information than they could with Access. Together, they decided to begin using the standalone Beamex calibration management software.
Shortly thereafter, in 2008, there was a corporate initiative to remove standalone software applications. Jody, IT and a Maximo contractor were tasked with replicating all of the Access application functions along with instrument and control data into the corporate work management system, Maximo. It was a difficult project, but the team was able to duplicate most of the functionality they used in Access. Major accomplishments included more secure data, improved calibration reporting capabilities, less stressful state audit reports, links to drawings, documented DCS points, asset data comparison, a better governance process, and more detailed instrument meter data. However, there were still inefficiencies, costs were high and significant manual oversight was required.
In 2012, Jody became the generation Business Unit Representative with the responsibilities of leading the generation team and interfacing with IT to implement Maximo and SAP. This project partnered the instrument shop’s valuable experience with the IT developers to meet business requirements. Maximo was upgraded from version 5.2 to 7.5, which included a new “built-in” calibration module. The technicians tried using the Maximo calibration system, but they pushed for CMX to be integrated into Maximo because they needed a calibration software that offered more comprehensive features.
Determining the needs of the entire business
By then, Jody had progressed to Business Analyst and the job implementing a calibration process change landed on his desk. He was tasked to give a recommendation on whether or not CMX could integrate into Maximo 7.5, as it needed to function to meet the business requirements. Jody knew that he could not make this decision without some thorough investigation, and he could not make it alone.
This integration project would require a substantial amount of planning and input from a team of internal departmental experts to ensure that it functioned appropriately for all parties involved. Also the external parties, or vendors, would be just as vital to their success.
Jody began by researching IT integration projects. He was soon amazed to discover the mind boggling number of failed projects, costing companies up into the trillions of dollars. He read about major failures where no progress was made, even situations in which companies were forced to go back to the original way after failed attempts. He declared, right then and there that, “failure is not an option.” Through a preliminary analysis, he concluded that this integration project would require a substantial amount of planning and input from a team of internal departmental experts to ensure that it functioned appropriately for all parties involved. He also knew the external parties, or vendors, would be just as vital to their success.
The Project Team:
Business Technical Lead (1)
Business Process Analyst (1)
Integration Team (2)
Site Manager (1)
Maximo Team (1)
Site SME (1)
I.T. Manager (1)
It was important that he put together a quality team that he trusted, because he knew he had to rely on everyone’s input and expertise. During this process, he learned important lessons about building a successful team. Jody soon discovered how each party tended to speak different technical languages as well as have different goals and ideology. He determined that communication was going to be the key to success. Jody explains, “the business will say they need an apple cut in 6 pieces and the IT side will hear cut a watermelon in half. Technical, cultural and language communication barriers are real challenges that needed full attention.”
He knew they would run into many implementation roadblocks if the team did not work together during the entire process. The team stayed focused on the detailed requirements and met often to review the business expectations.
Responsibilities of vendors and customer
As important as it is for the entire project team to understand everyone’s roles and responsibilities to ensure efforts weren’t duplicated or missed altogether, it was also essential to define the roles of the vendors and establish clear operation guidelines. The responsibilities and some of the sector’s key duties were:
- Business process
Data integrity is an important and an ongoing process. For SRP, it has never stopped since it first began in 1974. It is a time consuming, but important process – one which can go south in a very short period of time if it is not continually monitored. SRP put a lot of man hours into ensuring clean data.
- Calibration process
- Data conversion
- Integration support
SRP relied on Beamex’s expertise. Beamex acted as consultants and were quick to communicate how the integration could work most efficiently and made no empty promises.
- Datasheet conversion
- Maximo code
The Maximo team worked hand in hand with SRP technicians to meet business expectations and functionality requirements.
- File transfer to/from CMX and Maximo
It was imperative to make sure the right data was transferred back and forth between systems in the correct manner.
After analyzing all of these factors and gathering information from the project team, risks had to be considered so that Jody could be 100% confident that the integration would be successful. After all, failure was not an option.
How it works today
Upon completion of in-depth analysis by the team, Jody determined that the integration could be completed to meet both the business and IT needs. As Jody eloquently puts it, “it’s extremely simple, if you think of how complicated it could be.”
These are the basic rules used to form SRP’s system:
- CMX is the calibration system of record that stores the detailed calibration information.
- Maximo tracks all plant assets and is the master of scheduling.
- As for calibration, the only information Maximo needs is if an instrument passed or failed during the calibration event.
- In Maximo, there are two types of instrument assets. The first type are regular instrument assets that are never calibrated, for example an orifice plate. Secondly, there are calibrate-able assets, for example a transmitter.
- For a Maximo asset to be transferred into CMX, the asset has to be defined as a calibrate-able asset. Out of 28,000 instruments, there are 7,700 assets that require calibration and meet the calibrate-able asset criteria.
- If a Maximo work order is written or automatically generated by the preventive maintenance application for a calibrate-able asset, it automatically flows into CMX. This is critical because the rules create a built-in method of security that does not allow “garbage” data to be transferred back and forth. This ensures good data integrity for both software platforms. If a work order is not for a calibrate-able asset, it does not go to CMX.
- Work orders are generated by a planner. Technicians will paperlessly pick them up and calibrate them. This process allows field personnel to work only within CMX, and they do not deal with work orders in Maximo, saving them time, money and frustration.
For example, during a typical unit overhaul, many of the site’s 7,700 calibrate-able instrument assets need to be tested. Work orders are planned, put into progress, the information is automatically transferred to CMX and the technician is alerted by the planner via email. The technician can then download the asset test information to an MC6 documenting calibrator and perform the necessary work. Since the MC6 is a multifunction, documenting calibrator, the entire calibration process is automated because the results are stored in the calibrator’s memory. When the technician returns to the shop, they upload results into CMX. When a calibration test passes, an automatic notification is sent back into Maximo that closes the work order and documents who performed the work and when it was done. A failure requires the initiation of a follow up work order.
Throughout this process, Jody notes some key factors he recommends to keep in mind when implementing a calibration process change:
- Do not compromise data integrity.
- Build a solid team.
- Set realistic timelines.
- Set expectations and interpretation guidelines.
- Document the business process.
- Build a governance process.
- Support the new process.
Summary and the results
Salt River Project’s calibration processes have evolved tremendously over the past 40 years. As new technology solutions were developed, leaders, such as Jody, demonstrated key insights to embrace the advancements while balancing the necessary changes to create tailored work processes for SRP’s business needs. SRP has not only benefited from investing in quality solutions, but from doing their due diligence to carefully plan out the implementation of new processes.
“We were able to realize a significant return on investment during the first unit overhaul. It’s unusual, since ROI on software projects is usually nonexistent at first.”
Close attention to detail and building a trusted, quality project team (both internally and from their vendors) were crucial factors to SRP’s many successes. As a result, as Jody explains, “With this software integration project, we were able to realize a significant return on investment during the first unit overhaul. It’s unusual, since ROI on software projects is usually nonexistent at first.”
The most significant impact overall is that Salt River Project has been able to save about 30 minutes per calibration using an automated approach. This equates up to 1,000 man-hours in the previously cited unit overhaul example. Further savings are anticipated as history analysis will confirm that extended calibration intervals are recommended. It is important to note that SRP’s work order history for calibration is 100% automated and technicians never work in Maximo.
Other major benefits of the automated calibration system include:
- System oversight has been minimized.
- Audits are easy to perform and are less stressful.
- Defined calibration procedures provide a corporate “best practices” approach to calibration.
- Better decision making because of accurate data.
In the simplest terms, the new Beamex/Maximo calibration system gives back time to the people working in the field.