Posted on: 24 August 2018    

I always enjoy the opportunity to learn from others in my industry and to present my latest work, and this year at the New Zealand Esri User Conference (NZEUC) was no exception. I like getting technical, diving deep into problems, and evaluating and developing a solution. I found that the NZEUC was somewhat missing this technical aspect, and I thought that if I was disappointed by the lack of it, then others may be as well. And it seemed I wasn't alone...with the room at full capacity, I delved into the intricacies of moving data between disconnected environments, automating the deletion and creation of services and extending the capabilities of the out-of-the-box ArcPy tools.

My presentation was based on work that I have undertaken at the North Canterbury Transport Infrastructure Recovery (NCTIR) Alliance. My secondment to NCTIR began 18 months ago and has allowed me to develop and automate a number of procedures and systems. These are focused around the ability to automatically create and update services, and are further complicated by the setup of our infrastructure.

The only way we can communicate and transfer data between our ArcGIS Enterprise (Server) setup (Azure) and the rest of the NCTIR infrastructure (Datacom) is via FTP. To automate this process  I developed a custom python based process that hooks into our FME process with a custom transformer. The process performs a few key steps:

  • Zips up the GDB
  • Calculates the checksum and stores this along side the relevant service(s) in a separate file
  • FTPs both of these to the server
  • The server receives these and validates and extracts the GDB
  • The service(s) is  stopped and the old GDB is over written with a new one

This takes no more than a few minutes (dependent on file size) and has allowed us to automate the updating on many large datasets that were not suitable to exist in SDE.

A similar process is also used to automate the creation of Image Server services. The survey team at NCTIR perform and manage all the UAV Surveys, these are then stored in the local shared folders. I developed a ArcPy process to take these raw images and publish them as an Image Service. The script(s) performs the following steps:

  • Create a mosaic dataset and add the raw imagery
  • Calculate statistics, pyramids and overviews
  • Builds an image cache
  • Transfers all the data (cache, overviews, GDB, raw data) to the server
  • Creates a new service, using Regular Expression to alter the properties

These scripts run on demand when new imagery is added, and has resulted in a considerable time saving for the GIS team.

If you are interested in learning more about these processes, please don’t hesitate to get in touch with Hamish.

Blog written by Hamish Kingsbury, Senior Spatial Data Specialist, Abley.

Hamish IMG 7917

Hamish formal grey high