Animation powerhouse Illumination Mac Guff increases output using Avere Systems’ FXT Edge solution

As a computer graphics (CG) animation house, Illumination Mac Guff places heavy demands on its storage arrays. When parent-company Illumination Entertainment began working on animated feature films in 2004, it only produced one animated feature film per year. As Illumination Entertainment’s motion picture slate has grown to two to three films per year, so, too, must its production capacity.

With major features such as the successful Despicable Me franchise and the Dr. Seuss story The Lorax in the offing at that time, the video rendering workload to produce realistic animated effects were considerable. In order to meet the rapidly growing demand that the ambitious production schedule placed on the storage infrastructure, the company needed to address the disconnect between the number of spindles in its storage network, the volume of work, and the performance. It also wanted to pursue a vendor agnostic approach to ensure it could satisfy the production needs without being locked into one manufacturer.

Bruno Mahe, head of technology at Illumination Mac Guff, considered several solutions to solve these problems and ultimately chose Avere Systems FXT Edge Series filers. It was not only the increased throughput that the Avere solution offered – solving the I/O problem without adding more spindles – but also the real-time visual analysis that the system provided.

“It was the analytics software’s ability to show where the hot data lies within the storage system as a whole that really impressed me,” Mahe said. “I didn’t know before we did testing that it would be such a tremendous help to production. It was one of the things that sealed the deal.”

Avere’s partner-led approach

When it came to installation, French reseller NewNet was called in as the storage specialist that had introduced the existing Hitachi Data Systems (HDS) BlueArc storage arrays to Illumination Mac Guff. The VAR worked with Avere to fine tune the system and balance the workloads placed on the storage.

Before the installation of the FXT Edge nodes, the HDS array was under tremendous strain, especially during the day when the team of artists were busy weaving storylines into keyframes, and the 15,000-core rendering farm was churning away to render the fine-grained, subtle details of light, shade and movement to bring the images to life.

After the NewNet team and Avere experts completed their installation, the system was brought online and there was a dramatic change. Chic McGregor, EMEA, sales director at Avere, commented, “The effect was immediate. Before we had deployed our devices, there were multiple flashing lights indicating heavy disk utilization, but afterward there were only occasional blinks from the HDS drives because we’d offloaded so much I/O from them to the Avere system.”

Analyzing the problem

The next stage was to get the loading right to create the most efficient balance between the differing requirements of the artists and the rendering processes.

“The artists were using less resources storage-wise than the rendering farm, but they were more sensitive to resource contention. If they had to wait half a second to access the storage array, it would create a huge productivity issue. On the other hand, a 24-hour rendering process wouldn’t be affected too much if it took a few milliseconds more to gain access to a texture file,” said Mahe. Positioning the FXT edge devices in front of the storage arrays meant the FXT cluster could see everything passing across the network.

The analytics software played a major role by visualizing the flow and instantly highlighting where I/O contention bottlenecks were occurring. Rather than trawling through columns of figures, relying purely on experience, and playing hunches, the installers could see an I/O profile, allowing them to apply their skills more efficiently. This sped up a successful implementation by eliminating trial and error processes, saving time and money.

Mahe said: “Avere helped diagnose where the issues lay in our storage infrastructure, whether it was in the storage stack, in the network or elsewhere.”

Prior to Illumination deploying Avere, there had been a network connectivity problem with the HDS equipment. Mahe said there had been no way to isolate such an important part of the network to investigate the problem without interrupting work.

The diagnosis and resolution took two weeks and involved building a “sandbox” to reproduce the problem to allow access to the data flow without interrupting the daily production processes.

“If we had Avere and its analysis tools at that time, the two week problem would have been resolved in two minutes by just looking at the GUI,” he said.

Solid solution for cost-cutting

The offloading of the I/O from the disk system to the FXT Edge filers has resulted in a cost savings by avoiding frequent upgrades to the storage system. Before Avere, every time the storage array disks reached capacity, new high-performance storage hardware had to be purchased. The balance between operational SAS drives and near-line disks for archiving also had to be adjusted. Now that the Avere solution is in place, they take care of the I/O and the only change in the hard drive storage Mahe has planned is for near-line storage, which will be a cheaper, slower tier of storage. He now feels that the existing array could be replaced – but that depends on what the storage company’s next move may be.

Avere FXT Edge filers are built using RAM and solid state drives (SSDs), as well as high speed hard disk drives (HDDs), employing automatic tiering across both the Edge filer cluster and other vendors’ storage arrays or core filers. This accelerates performance and offers the opportunity to reduce costs. As the number of network-attached storage arrays increase, they are seen as a single, self-contained logical unit by the Avere management system and from the clients’ viewpoints.

Flash of inspiration

Illumination Mac Guff depends upon a tiered storage approach for its data strategy. Even when a film has been released, the company has to keep movie data in near-line storage to facilitate requests for promotional material.

This has led the company to track Avere’s development of its FlashMove®and FlashMirror® capabilities with great interest. Mahe explained, “Right now, when we move files externally between storage tiers, we have HDS for the main production, another storage vendor for near-line archive, and then there’s an even slower system for static files. The best tool we have had is the HDS duplication feature. It’s great, it’s reliable, and it’s fast, but we can only use it with HDS equipment. Avere FlashMove and FlashMirror offer alternatives to this as they are a vendor agnostic replication toolset.”

Overall, Mahe says that he could never go back to the old system of adding additional disks to a traditional storage system to scale performance, it just would not work and it’s cost prohibitive. He now has a working environment that suits all the players, plus the ability to see everything that is happening within the storage environment through the analytics capability provided by Avere – a real bonus.

“We used I/O profiling before, which was client based, but we didn’t have the overview of the complete storage network that Avere has given us, which is proving to be exceptionally helpful. It has shown us hot spots in real-time, which has been a major help with production. Now we have this real-time looking glass, which does not impact production and that’s of great value.”

The improved performance of Avere’s FXT Edge modules and the ability to continually monitor and adjust throughput using the visual analysis tools has resulted in a significant increase of throughput for Illumination Mac Guff for a relatively small expenditure outlay.

About Illumination Mac Guff

Illumination Mac Guff is one of the world’s leading studios for animated feature films. Based in Paris, the company is known for such global blockbusters as Despicable Me, Despicable Me 2, and The Lorax – all produced Illumination Entertainment for Universal Pictures.