Category Archives: Geoprocessing with Python

Jun 24 2011

Python geoprocessing with .csv and field types


A while back I showed how to import a .csv to create a shapefile.  The script that I had written for a client went unused for quite sometime and after trying out the resulting shapefile it was determined that the fields from the .csv were being formatted incorrectly in the resulting shapefile.

Last night I set out to fix this.  The idea was to create a blank database table with the proper field formats and then copy the values from the .csv to the database table.  What I thought was going to take 30 minutes to complete, turned into a long adventure.  About 6 hours later I finally figured it out.  Interestingly enough, I found almost NO info on what I was trying to do.  Finally I found a partial reference in a forum thread from 2006 and wasn’t sure if it would even work.

# import required modules
import sys, os, glob
import arcgisscripting

# Create the geoprocessor object
gp = arcgisscripting.create()
# I was having issues with a table view not deleting so I added the overwrite
gp.overwriteoutput = 1

## System arguments
# Input .csv
File_in = sys.argv[1]
# Output shapefile
File_out = sys.argv[2]
# Input table_template
tabletemp = sys.argv[3]

# Set the workspace for the processing files
gp.workspace = os.path.dirname(sys.argv[1])

# Convert the .csv to .dbf in the directory from sys.argv[3]
gp.AddMessage("\nCreating processing .dbf from .csv...")
gp.TableToDbase_conversion(File_in, os.path.dirname(sys.argv[3]))

## Make copies so that processing doesn't take place on the originals.
gp.AddMessage("\nCreating processing file...")
gp.Copy_management(File_in[:-4]+".dbf", "tempfile.dbf")
gp.Copy_management(tabletemp, "temptable.dbf")

# Need to use a search cursor to read the values from the input file
# and then use a insert cursor to write to the new table.
# The part I could not figure out is the SetValue function.  By using a
# while statement we can read thru each of the records in the input .csv
# and then write them out using SetValue.
# The SetValue takes two parameters; the first is the field name in the new
# table and the second is the field name in the read file.  Quite similiar
# to Field Mapping actually.  The input field name must not match the output
# field name, however the formats do need to match.
gp.AddMessage("\nSearch cursor process...")
intable = "tempfile.dbf"
outtable = "temptable.dbf"
table = gp.SearchCursor(intable)
run = table.Next()
outRows = gp.InsertCursor(outtable)
while run:
	outRow = outRows.NewRow()
	outRow.SetValue("FieldName1", run.FieldName1)
	outRow.SetValue("FieldName2", run.FieldName2)
	outRow.SetValue("FieldName3", run.FieldName3)
	outRow.SetValue("FieldName4", run.FieldName4)
	run = table.Next()
del outRows

# From here you can do whatever you wish with the new table "temptable.dbf"
# If you want to make a points shapefile you will need to use the MakeEventLayer
# as shown below.

gp.AddMessage("Creating the output points file...")
# Make a point shapefile based on the X,Y in the table
gp.MakeXYEventLayer_management("temptable.dbf", "X", "Y", "Table_output")
gp.CopyFeatures_management("Table_output", "temp.shp")
# Create a prj file for temp.shp
coordsys = "Coordinate Systems/Geographic Coordinate Systems/World/WGS 1984.prj"
gp.defineprojection("temp.shp", coordsys)

gp.AddMessage("Creating final output file...")
# Create the output files based on the user's input
gp.Rename_management("temp.shp", File_out)

gp.AddMessage("Cleanup...")
# Delete "process" files
for file in glob.glob(gp.workspace+"/temp*"):
    os.remove(file)
for file in glob.glob(gp.workspace+"/*xml"):
    os.remove(file)

gp.AddMessage("\nImport Successful.\n")

Hope this helps!
BR

Apr 14 2011

Python Geoprocessing


Alright everybody, I get a lot of searches everyday for Python geoprocessing related questions.  I would love to add more content about geoprocessing but not sure what it is that all of you would like to learn.  I was going to do a “Geoprocessing tip of the week” but I felt that it would be pretty much the same as looking at the ESRI help docs.

The simplest way is to email me via the about/contact page and I will attempt to answer what I can.  I am also looking at putting on a Python Geoprocessing class(s) here in Phoenix at the DesertPy.org group meetups.  Would there be any interest in a Python Geoprocessing class if I were to have one?  Maybe if there is enough interest we can do a webinar so that those outside of the Phoenix area could also participate.

As always, feel free to drop me a line…

BR

Feb 13 2011

Creating a simple Python Geoprocessing tool and adding it to ArcToolbox in AV 9.3


Ask and thou shalt receive… Well it seems as though there is a LOT of interest out there in Python Geoprocessing for ESRI ArcView 9.x.  This increased interest may be due to the release of ArcView 10 with ArcPy.  For those of you that do not know, Python has been shipped with ArcView for many years now.  The ArcToolbox tools are based on Python and when you create a model with the model builder you can export the Python code.

This week’s tutorial will be creating a very simple Python geoprocessing script and then creating a new ArcToolbox tool and adding the Python script to the new toolbox.

I once thought it would be nice to have a simple tool that told me all of the field names in a shapefile.  Sure I could open up the attribute table and see them but if I did that I wouldn’t have written my simple tool.

The Script

I think this script is simple enough to explain it with remark statements.


#listfields.py
import sys, os, glob

# We need to create the geoprocessor object
import arcgisscripting
gp = arcgisscripting.create(9.3)

# We need to create an argument and assign it to the value of "file"
file = sys.argv[1]

# The workspace always needs to be identified and here we will use the location of "file"
gp.workspace = os.path.dirname(sys.argv[1])

# A good friend to use is the AddMessage.  This will print to the tool window.  Very useful with troubleshooting as well.
gp.AddMessage("Starting Process.....")

# Using ListFields, we get a list of the fields in the inputted "file"
fieldlist = gp.ListFields(""+file+"")
for field in fieldlist:
    FieldName = field.Name

    # As we go thru the list, AddMessage will print each field name out to the tool window
    gp.AddMessage(FieldName)

# And we add one last message so that we know the entire process completed
gp.AddMessage("Process Complete")

Pretty simple stuff there, but I just wanted to use a simple script for this tutorial.

The Toolbox

Now we will create a new Toolbox where the list field’s script will go.

  1. In ArcToolbox, right click on the ArcToolbox folder at the top and select New Toolbox…


  2. Give your toolbox a name


  3. Right click on your new toolbox and select Add => Script


  4. Fill in the fields on the Add Script and click “Next”. I always use the “Store relative path names…..” to make sharing the toolbox easier.


  5. Select the listfields.py file and check the “Run Python script in process” then click “Next”


  6. Now the hard part. In the script that we created, there was one argument that we used “file = sys.argv[1]” The beauty of Python geoprocessing is that there is a user interface to work with and you do not have to write all of the requirements for your inputs or output.
    1. The display name is what you will see in the tool. For this example we will use “Select your file:”
    2. The Data Type produces a drop down to choose what type of file is to be selected. For this example we will use “Shapefile”


    3. The default Parameter Properties will all be correct for this example. These properties are used to control each of the Parameters (in this example we only have the “Select your file:”) Settings vary from Required or optional field, input or output direction etc.. Click “Finish”


  7. That is all there is to it. You now have your new toolbox entitled “My tools” and within that toolbox a new tool entitled “ListFields”


  8. As with any Toolbox tool, click on the new tool to open it:


Go ahead and select a file and click “OK”.

Obviously this is a basic example of what can be done but it is a start.

Feb 5 2011

Welcome to my blog…..


I started to learn Python programming last year to use in ArcGIS.  I read a couple of books about Python and jumped right in.  I had help in the beginning from an avid forum user.  Since then I have continuously ventured into new areas of using Python.

Recently the user forum at ESRI for Python Geoprocessing underwent some changes that makes it hard to sift through the posts there.

As I started to take Python outside of ArcGIS, I was immediately confronted with a lack of information on wxPython and using geoprocessing.  With a lot of help from friends I am well along in creating a GUI that uses wxPython as well as incorporating ArcGIS geoprocessing.

My goal for this site is to continuously add script examples and to create a meeting place for Python users.

I recently started to add articles about route optimization or Solid Waste routing.  I just wanted to explain the reason that I added this onto my otherwise all Python blog.

It is because of my job doing Route Optimization that I ever ventured into Python and the rest that followed.  Our primary software runs on top of ESRI’s ArcView.  From always using ArcView, I found a need (or more of a desire) to automate some of my daily functions.

As I have learned from programmers, it seems that the majority of the programmers that I know have the ideology that the least amount of work that they need to perform, the better.  This isn’t to say that they are lazy, this is to say that their goal is to automate as much as possible and work smarter not harder!

So Python was the natural way to go for me.

Although the company that I work for sells Route Optimization software, I felt the need to share my knowledge of “alternative” techniques to route optimization and/or improving efficiency in Solid Waste collection.  I plan on adding more posts regarding this in the future.

So even though Route Optimization itself has nothing to do with Python, it is what sent me down the Python Path.

Jan 30 2011

Export Shapefile as .csv tutorial


I recently had issues with a simple export to .csv. What I wanted to do is pull in a shapefile and delete a bunch of fields and then export it out to a .csv (this works the same for saving to a text file). It actually turned out easier than I thought as I did not need to delete all the fields but rather I just wrote out the fields that I wanted to keep:


# Import system modules
import sys, string, os, glob, csv
import arcgisscripting
# Create the Geoprocessor object
gp = arcgisscripting.create()

# System arguments
input = sys.argv[1]
output = sys.argv[2]

# Set our new outputnew to be opened output with write privileges.
outputnew=open(output,'wb')

# Set linewriter as csv.writer written out to outputnew
# The delimiter can be set as comma, pipe etc.
linewriter=csv.writer(outputnew, delimiter=',')

# If you want to pass all fields thru then disregard this.  I use this as I only want to pass some of the fields out and these are listed in Python list
good_fields = ['field 1', 'field 2', 'field 3', 'field 4', 'field 5']

# Now we go thru all of the fields and keep just those that are in the list of good fields
flds=gp.ListFields(stops_in)
fld = flds.Next()
header=[]
while fld:
    if fld.Name in good_fields:
        value=fld.Name
        header.append(value)
    fld = flds.Next()
linewriter.writerow(header)

# And we run thru each row populating the kept fields
row=cursor.Next()
row_num = 1
while row:
    line=[]
    if row_is_valid(row_num, row):
        for fld in header:
            value=row.GetValue(fld)
            line.append(value)
        linewriter.writerow(line)
    row=cursor.Next()
    row_num += 1

del cursor
gp.AddMessage("Finished")
output.close()


I originally attempted to make a copy of the original shapefile and then a table view from the shapefile. In the end all I needed to do was simply read the input shapefile and then write it out to the .csv file.

Jan 30 2011

Adding a Python Script to a new Toolbar in ArcView 9.3.1. tutorial


So you finished your Python ArcToolbox tool and now you want to add a shortcut to the toolbar. This tutorial is based on the assumption that you have already added your script to the ArcToolbox tools. If not I will cover that in another tutorial. I will show how to add a button in addition to how to add a menu to ArcView toolbar. Both are convenient ways to launch your Python script tools without having to open ArcToolbox.

First we will create a button to launch your tool.

  • Go to Tools > Customize


    • This will open up to the Toolbars tab. If you do not want to add your own Toolbar, skip this section.
  • Click New for a new toolbar:


  • Name your toolbar
  • Save it to either the current project (the .mxd) or to the Normal.mxt for use in all projects.
  • This will create an empty toolbar. Drag the toolbar to the top of ArcView.


Now that you have a new and empty toolbar we need to create the button for the Python tool.

  • Click on the Commands tab under Customize:


  • Scroll to the bottom of the Categories window:


  • Highlight [UIControls] and at the bottom click New UIControl


  • Select UIButtonControl and hit Create
  • Click in the box for the name if you wish to rename the tool


  • Next, click and drag the new UIButtonControl1 to your toolbar that was just created.


Now that the button is one the toolbar, we need to set it up to launch your Python tool.

  • Right click on the new button and you should see several options here.


  • You can change the button image to whatever you would like or you can just use “Text Only” for the name of the tool.


Now that we have the button on the toolbar and changed the way that it displays, it is time to add the Visual Basic snippet to launch the tool.

  • Click on “View Source”. This should open up the Microsoft Visual Basic editor:


 

 

  • Copy and Paste the following script between the Private Sub UIButtonControl1_Click() and End Sub:

Dim pUID As New UID
pUID = "esriGeoprocessingUI.ArcToolboxExtension"
Dim pGP As IGeoProcessor
Set pGP = New GeoProcessor
Dim pATBExt As IArcToolboxExtension
Set pATBExt = Application.FindExtensionByCLSID(pUID)
Dim pAtb As IArcToolbox
Set pAtb = pATBExt.ArcToolbox
Dim pTool As IGPTool
Set pTool = pAtb.GetToolbyNameString("mypythontoolname")
Dim pCommand As IGPToolCommandHelper
Set pCommand = New GPToolCommandHelper
pCommand.SetTool pTool
pCommand.Invoke Nothing</span>
  • You will need to replace “mypythontoolname” with the name of your Python tool.
  • Click the save button at the top of the VB editor.
  • Click on your button and it should open.

There are a few things that can cause the button not to open your Python tool. The most common that I have seen is that the Visual Basic reference is not selected as available for using the ESRI GeoprocessingUI Object Library. This can be added by going into the Visual Basic editor again. At the top there is an option for Tools and then select references. Make sure that the ESRI GeoprocessingUI Object Library is checked. If not, check it to make the library available.

If you want to add a drop down menu for listing multiple Python tools the steps are mostly the same. After creating a new toolbar, you can create a new menu before you create the tool buttons.

On the Customize > Commands tab in the left hand pane there is an option for [New Menu]. Create the new menu in the same manner as creating the UIControls button. Add the menu to the toolbar and then you can add tool buttons to the drop down menu by clicking and dragging them to the drop down.

Hope that helps!

Dec 6 2010

Using text file to create Shapefile


Well, it seems as though all of the things we learn, are learned due to necessity.  I have always worked with either layers or database tables in my Python Geoprocessing scripts.  It has always worked so there has not been a need to try something different, until last week.

Working on a Python tool for importing a file I requested a .dbf table.  I kept running into issues with the formatting or the header names or what have you.  I was asked if I can take a .csv for an input file.  I assumed this was possible so off I went to solve my new issue.  After a couple of quick Google searches I found what I needed and the ESRI thread that I found was answered by a friend of mine named R.D. and if you spend ANY time on the forums, you will see his name.

CODE


import sys, os, glob
import arcgisscripting

# Create the geoprocessor object
gp = arcgisscripting.create()

# Create the geoprocessor object

## System arguments
# Input Customer .csv
Stops = sys.argv[1]
# Output Stops <a class="zem_slink" title="Shapefile" rel="wikipedia" href="http://en.wikipedia.org/wiki/Shapefile">shapefile</a>
Stops_output = sys.argv[2]

# Set the workspace for the processing files
gp.workspace = os.path.dirname(sys.argv[3])

import arcgisscripting, os
gp = arcgisscripting.create()
gp.AddMessage("\nCreating processing .dbf from .csv...")
gp.TableToDbase_conversion(Stops, os.path.dirname(sys.argv[3]))            #1
gp.AddMessage("\nCreating Event Layer")
gp.MakeXYEventLayer_management(Stops[:-4]+".dbf", "X", "Y", Stops[:-4])            #2

# Create a prj file for Stops_output.shp            #3
coordsys = "Coordinate Systems/Geographic Coordinate Systems/World/WGS 1984.prj"
gp.defineprojection(Stops[:-4], coordsys)

Obviously that is just a snippet of the code but you should get the idea.  #1 shows using the TableToDbase_conversion geoprocessing function.  I have found very little info on this function so I do not know what possible arguments that can be used.  What I do know is that you will get a .dbf file with the exact same name as the original file and the .dbf extension.  #2 Then takes the resulting .dbf from #1 by using the input Stops[:-4] to take the original name and remove the .csv (4 spaces back) and then concatenated it together with “.dbf”.  This then uses the X and Y coordinates to create the new layer and assigns it to the original Stops input name.

In order for the new resulting shapefile to work, you will need to define the projection to the correct coordinate system as seen in #4.

Hope this helps :)

BR

Oct 12 2010

Review of ArcView 10


Today I went to a seminar from ESRI for increasing productivity in ArcGIS 10.  I will say that I am very excited about the new functionality in AV 10.

Some of the bigger things that I noted:

  • ArcGIS Mobile is available with all AV10 licenses
    • Available for Apple IOS
    • Will be available for Android by year’s end
    • Allows users to input data in field
    • It is available for ArcView Desktop
    • ArcGIS Mobile
    • Basemaps
      • Available online with ESRI global account (free)
      • Shared data from all over the world
      • Imagery to 1 meter or better in U.S.
      • These are community base maps available
  • Can create basemaps  within AV10
    • Add layers by dragging them in and it creates like a cached layer
    • Refresh is almost instant for the basemap
    • Layer Package
      • They have a new layer package so that you can send layer(s) to others but in the package it maintains the theme definitions in addition to the symbolog
    • ArcGIS Explorer
      • Free for web use
    • Can create slideshows
  • Parcel Management Toolbar for editing parcel data
  • Data reviewer can be setup to check for user specified issues
  • Automated Map Books
    • Supported by ESRI now
    • Created by PyArc script
    • Creates book pages and cover/index pages
    • Geoprocessing runs in background so that user may continue to work while tool is running
  • ESRI is on Amazon EC2
    • Can easily create ArcGIS server on cloud
    • There is a search function to create a server for different preset configurations
    • License borrowing in AV10, can transfer temporarily to another user.