PLSS Research & Data Restructuring - Restoring a County GIS (Part 6)

Map of the Public Land Survey System - US Dept of the Interior, Bureau of Land Management

The next phase in restructuring the County data towards dependency on a more solid framework of geographic data consisted of building out the Public Land Survey (PLSS) layer. As noted in the previous chapter, the essential part of this work consisted of collecting Ground Control Points and Registered Land Corners for addition to their own respective layers.

Assembling all of the point data related to the Land Corners layer, numerous inconsistencies were noted, which required further investigation to understand their relevance to the situation.

Section corners appeared to have multiple locations along the eastern edge of the county, as well as along old Kansas City boundaries. Additionally, limited availability of data related to the boundaries along the river created challenges on that front, requiring in depth research to understand all of these issues and their relationship to one another.

. . .

Downloading the PLSS layer from the Missouri Spatial Data Information Service (MSDIS) answered a few questions, but begged clarification on others. Partial sections adorned the eastern boundary of the County, and of the same Township/Range/Section as those adjacent in Clay County. What caused this?

The part of the County along the Missouri River valley had its own set of problems too. Some of the Township/Range information seemed to be completely wrong. Township/Range composed of 50s/30s dominated the County, but in this part of the County, 7/20s popped up along the river. Where did these come from?

While researching these questions, I came to know University of Missouri Professor and co-author of the “Atlas of Lewis and Clark in Missouri,” Jim Harlan. He developed the PLSS layer I downloaded, was very informative, and just a lot of fun to talk with. Validating much of what I already understood, he clarified a few other points as well.

The Platte Purchase of 1836 caused the issue along the eastern boundary. I am still not quite sure that I understand why the surveyors did not just pick up where they left off when they began their survey; instead, a slight jog remains today along the Platte County Boundary, where partial Sections of that part of the Township / Range meets partial Sections from the very same Township / Range in Clay County.

A lengthy technical discussion with Mr. Harlan did not exactly clarify the reasons behind the Sections mismatch, but did help me to understand the origin and impact. He attributed most of the issue and many other discrepancies I would soon uncover to excessive drinking on the frontier survey crews. Amusingly, it only took a little bit of reading to validate those conditions.

More easily understood, the sections along the Missouri River are a simple result of nature, which slipped my mind. Survey boundaries never move. The river does, and quite often. Most of Missouri is surveyed under the 5th Principal Meridian, while Kansas was surveyed using the 6th. Since that time, the Missouri River has wandered about quite a bit, leaving bits and pieces of what some perceive as other states on both sides of the river. This is most evident along the Mississippi River, but the Missouri has its own share too.

At least two mysteries were resolved, which only left the northern boundary of the County. That turned out to be a rather simple matter really. Everything was normal. The County boundary moved a few years prior, and several surveys were available, along with extensive documentation and precise GPS positions along the entire boundary.

. . .

Following the framework identified in the previous chapter, I started construction on what would be four distinct layers. This actually took quite a bit of time, as I originally thought that I might just put it all together in one layer, but revelations associated with the Parcel Number, forced me down another path.

The Parcel Number identified the location of the land, indicating the Township/Range, the Section, the Quarter Section, as well as a couple other elements; indeed, the first 4 numbers in the set represented the legacy Map Sheet # that any parcel with that number could be found on. The second part of that number caused a little difficulty, but a little research uncovered that it was an old National Weather Service thing that has somehow found its way into the Parcel number; it covers 4 sections.

Example Parcel Number: 18–9.0–32–400–015–001.000

• 18 = Township 52 North / Range 33 W — 5th Principal Meridian
• 9.0 = Tetrasection 9.0
• 32 = Section 32
• 400 = Southeast quarter
• 015 = Block Number
• 001.000 = Lot . SubLot Number

Taking the Land Corners feature previously generated, along with the PLSS from MSDIS, I began tying corners to points. It was strictly a Section PLSS, and most of the corners on the PLSS did not move far, usually only a handful of feet in just about every direction. There were a few exceptions, and I decided it would be best to set those aside until completing more research on the specific attributes of those.

Once complete, I began developing the framework for the PLSS that would live on our system. It would be comprised of four independent layers. The current Map Sheet number already in use would carry through all layers.

The PLSSTownships defined the Township/Ranges for further division. PLSSFirstDivision was the first division of the Township/Range layer, and contained the Tetrasections. PLSSSecondDivision was the second division of the Township/Range layer, and contained the Sections. PLSSThirdDivision was the third division of the Township/Range layer, and contained the Quarter Section information.

Fields in each of the layers were named straightforward and simple, but still abbreviated. When developing these, I discovered that while you could give field names any number of characters, if you went to export any of them to a Shapefile the names were truncated. Consequently, all of the field names needed to be eight characters to provide clarity and prevent any confusion about the name.

Most of the data was numeric, or simple text. This attribute information would rarely, if ever change. The field definitions would always work, regardless of what changed though. I used Short Integer everywhere possible, as well as limiting the length of text fields. In the case of cardinal and ordinal directions, I simply created a domain (lookup table).

The layers do not contain all of the data for each of the division. They only hold that relevant to the County, which causes it to appear as if data is missing along the boundaries. There was just no need to define Sections that were not in Platte County though.

We left the boundary on the Missouri River alone. There was just no help for that. Surveys were more than 100 years old in some cases, or had never been properly located with GPS.

GPS positions identifying County boundaries on the north and east side of the County enabled us to re-define the boundary with 99% confidence. The work added hundreds of acres to County that had not been previously included or assessed, resulting in increased revenue for the County.

In the end, we only used the PLSSThirdDivision in our production environment. It had everything we needed in it. All of the other layers ended up being really for other cartographic purposes only and for analysis, and quality control. I simply made all adjustments in the PLSSThirdDivision layer, and migrated those changes up to the other layers once a month.

Building them out independently proved to be a good exercise that provided valuable insight when I started building the Subdivision framework. It helped in understanding the numerous peculiarities of working with DB2, ArcGIS, and ArcSDE, when building out new features and tables. It also provided an excellent understanding of the idiosyncrasies of the Public Land Survey System.

We decided not to involve too many parcel adjustments at this stage. There was more work to do and another framework to complete. New parcels that did not belong to subdivisions were subject to the new PLSS, and adjacent parcels adjust out of the way of them. Additionally, the individual receiving deeds in the Office of the Assessor began updating all associated parcels, where previously they had only reviewed the legal to determine location. This worked well for the remainder of my time there, and just updating as it came through resulted in updating nearly 75% of parcels in the County over 5 years.

All individual features were stored in a Feature Dataset named Public Land Survey System, along with the LandCorners and GroundControlPoints, due to their geographic relationship and dependencies.

. . .

Map displaying all layers of the grid and Land Corners

Below are links to access the four new features created in this process. Full detail on the construction of these features is further defined within the Info Sheet, containing the Metadata, for each individual feature.

. . .

PLSSTownships feature (KML)

- Info Sheet for PLSSTownship Metadata

PLSSFirstDivision feature (KML)

- Info Sheet for PLSSFirstDivision Metadata

PLSSSecondDivision feature (KML)

- Info Sheet for PLSSSecondDivision Metadata

PLSSThirdDivision feature (KML)

- Info Sheet for PLSSThirdDivision Metadata

. . .

Coming next in the series … more data restructuring, Subdivision, re-alignment, and other points of interest!

Data Restructuring and a New Framework - Restoring a County GIS (Part 5)

Data was the biggest issue needing addressed within the County GIS, as I am sure it is in most places; more especially at the County and small municipal level. In short, it was a disaster. Data was inconsistent, without structure, had no restraints to ensure integrity. There was no architecture. There was no maintenance or adherence to best practices associated with maintaining a geo-database, or any sort of database really.

The extent of the damage necessitated quite a bit of rebuilding and will need discussed through multiple postings. Irritated by the revolving door associated with GIS positions at the County, that is where the inconsistency began. The recent departure of the previous GIS Manager and GIS Coordinator at approximately the same time compounded the problem, along with generally uncaring and incompetent GIS analysts. One of the stories told to me, was of a previous GIS analyst simply guessing the location to draw new parcels. This individual manually drew parcels on top of parcels, vaguely resembling the original survey drawing, and only generally in the correct area. He got it in the right County, at least.

Defining a framework was the first order of business. ArcGIS was the platform, the architectural choice rather simple. I was already familiar with the title by Nancy von Meyer, “GIS and Land Records: The Parcel Data Model.” It was the first book purchase I made after accepting the position. I did not have any experience with Land records outside of academia, but was very interested in the topic, and that book turned out to be the perfect starting point. Indeed, it referenced the ArcGIS Land Parcel Data Model, which I quickly located on the Library of Libraries, the Internet. With a plotter at my disposal, I printed a gigantic poster of the model for further study.

Upstairs at home with a whiteboard, the poster, and a lot of spare time in the evenings, I examined this poster in detail and made extensive notes. It was not the most efficient structure or architecture, but that was easily remedied, drawing on prior experience managing data structures. The main thing that I found so astonishing was that the model recommended defining nearly every data type as a string. Why would anyone in his or her right Database Administrator mind do that, except when necessary?

After I worked everything out on paper, I set about defining the first task. Each of the layers recommended required a definitive framework from which to base any sort of delineation, and that framework required control points to justify and maintain its position and integrity. The Office of the Recorder had all of the records related to Land Corner surveys, but a solid reference would be needed to ensure horizontal and vertical accuracy. Accessing the Missouri Spatial Data Information Service (MSDIS), I acquired the most recent Ground Control Points layer, a recently published layer for the Public Land Survey (PLSS), and was ready to begin creating new GIS data.

There was no need to change anything in the Ground Control Points layer, so it was imported to the database as is. The Land Corners layer was another story. The documents at the Office of the Recorder provided most of the information needed to accurately locate each of the Land Corners; although not all. The State of Missouri had been promoting restoration and location of Land Corners for a few years, and some of them contained true GPS coordinates. Those restoration and re-establishments done prior to 2000 rarely had them though, but did contain enough information to plot them with relative accuracy. The older documents were no help at all though, as they relied on measurements from odd little landmarks like bushes and trees.

The final structure of the Land Corners layer for the Geodatabase was relatively simple and contained seven essential fields for each record, and customized to ensure minimal space usage in the database. CORNERTYPE uses a numeric value and a Coded Value Domain (i.e., a Lookup Table) to restrict the naming possibilities and ensure continued integrity; as well, requiring uniqueness in some cases, and not permitting <NULL>. Most other fields for this feature are numeric too; even that for the Missouri Department of Natural Resources (MoDNR) Document Number. It is only a number after all. There are only two text fields (CORNERINDEX and CORNERNAME) in this layer, since the data would be a mix of alphanumeric characters derived from the Corner Master Index Grid diagram (left). These fields were limited to the number of characters permitted though. More information about the file layout is in the Metadata. That was the final touch, to begin documentation on all of this new and improved data and using Content Standard for Digital Geospatial Metadata as outlined by the Federal Geographic Data Committee. The final resting place for the feature would be within a Feature Dataset dedicated to Public Land Survey System data only and updated only when provided new information from MoDNR or associated filings by surveyors at the Office of the Recorder.

After creating the new feature, the only thing left to do was add the points through ArcGIS Map, using the data provided on the MoDNR document. This enabled me to move to the next phase of correcting the PLSS downloaded from MSDIS; that is for another post though.

Below are links to access the two new features created out of this process; the beginning points, as it were.

Ground Control Points Feature (KML)

- Info Sheet for Ground Control Points Metadata

Land Corners Feature (KML)

- Info sheet for Land Corners Metadata

Coming next in the series … more data restructuring and implementation related to the Public Land Survey System.

Untangling the GeoDatabase - Restoring a County GIS (Part 4)

Data was one of the biggest issues needing addressed at PCAO. The situation was a data disaster, but not just restricted to the data. Anything to do with structure, definition, maintenance or application of best practices associated with maintaining a geo-database had been ignored; all of this irritated by the revolving door associated with the GIS Manager role at the County.

About 5 years prior, ESRI representatives had come in and established the system for enterprise use and application and turned it over to the resident GIS Analyst of the time. Several had passed through that position in the years that followed; some less qualified than others; some attempting to maintain data and data structure integrity; others, not so much. The GIS Manager position had been vacant for about 6 months, prior to my arrival. 

The GIS Analyst during this time was a solid Cartographer but admitted having a very limited understanding of databases. She depended heavily upon the GIS Coordinator in the Planning & Zoning Department (P&Z).

The geo-database was a rather simple arrangement, leveraging ArcGIS SDE v9.1 on a IBM DB2 v8 database, providing unrestricted access to anyone that knew how. It contained all of the features used by the Office of the Assessor and some random features from P&Z. Some of the features had been copy/pasted from one feature to another, thereby duplicating the Shape_Length and Shape_Area field multiple times. 

Many other fields had also been duplicated also, from various attempts to manipulate the data fields All of the fields in all of the tables had been converted or were originally created as 256-character length varChar (text). It was a little hard to believe that ESRI would set it up that way, but I later found that was exactly the case. 

The geo-database also contained numerous incomplete features, and various types of analysis features, from unknown and unidentified projects. Nothing was documented. Meta data was missing from everything. Field names were nonsensical. There was no rhyme or reason to the organization or naming convention of many of the features or data fields.

Straightening out the situation was rather simple and only took about a month. I exported everything to a File Geo-database (FGDB) for analysis and permanent archiving, then removed anything from SDE for which there was no accounting. P&Z was the only other department storing data in SDE, and the Assessor applications were straightforward in their use of the data.

Once all of the features were exported to a FGDB, I carefully analyzed the duplicated fields to determine that containing the most accurate information. In most cases, the newly duplicated field had been used for data storage, from the point it had been duplicated; in other cases, the field and the data were duplicate fully. After completing the analysis, and removing all duplicated fields, and then restoring the newly scrubbed features to be retained.

The data that was to remain on SDE would be strictly for application use and data maintenance and not be stored without some sort of documentation about what it was and what it would be use for; thus, initiating a new and sorely needed policy for SDE.

First Environmental Steps - Restoring a County GIS (Part 3)

Upon taking on the position of GIS Manager, I spent the first few days familiarizing myself with the hardware and software currently in place, and what licensing was available on the same. I also took some time to meet with all of the various Department Heads and Officials I would be working with to get a better grasp of the political structures, boundaries, and my role in all of it.

The first of those meetings was with Information Services (PCIS). As a department of the County, it was made clear that while PCIS sometimes provided assistance to the Office of the Assessor (an independent office), but were not there for support. I was a bit surprised by all of this, and their apparent desire to continue that way, but knew that was not going to work out well. There was no way anything was going to get off the ground without some collaboration with them, since they were really the only other “technical” people in the entire organization. Through various means (and likely my constant badgering), we eventually threw that time-honored and highly restrictive tradition out the window.

Another issue that came up during that first meeting outlined one of the first tasks at hand. Asking whether there was any sort of network and database models available, they first looked at me as if I had a 3rd eye, and then asked what I needed that for. After explaining that it was going to be bit difficult to maintain this structure without knowing a little bit about how things were connected around there, they proceeded to sketch it out for me verbally. Nothing was documented. The problem did not simply exist in the department that I now managed, but was endemic in the organization. I later learned that nobody bothered to document anything for one rather simple and selfish reason; job security. I resolved to change that attitude; if nowhere else, within the GIS Department, hoping others would follow the example.

Since there was so much to be documented, I was not quite sure where to start, so began with what seemed the most natural place to start, an overview of the hardware and software in use. The workstations were still on Windows XP, running ArcMap v9.1; the geo-database residing on a Windows 2003 server, running IBM DB2 v8.1 and ArcSDE v9.1. The web server was on the same version of another server, running Apache Tomcat (don’t recall the version), and ArcIMS v9.1. In contrast, ESRI has just released v9.3.1.

The web application was a custom situation ESRI had assembled for the Assessor a few years earlier. It used frames, a bit of Java and Javascript to provide real estate information for the County, through a data or map search interface. This site was dependent upon ArcSDE, as well as loads created by views to data stored in the Collector, Recorder, Planning & Zoning, and Computer Assisted Mass Appraisal (CAMA) system for the Assessor. It worked relatively well with minor intervention, but sorely needed a facelift; if nothing else, to get rid of those frames. Unfortunately, the documentation for that was also missing in action; but, later found.

Licensing was quite fortunate. We held 2 ArcEditor licenses, ArcGIS for Server Basic (for the ArcIMS v9.1), and ArcGIS for Server Enterprise. I was pleased to see the later two and immediately brought it to the attention of the Assessor, explaining the capability. She was happy to hear, and equally disappointed that none who had held the GIS Manager position previously, had attempted to implement the ArcGIS for Server she had been paying licensing fees for. We decided to make an effort to make that happen.

Finally, a Standard Operating Policy & Procedure document was needed immediately, to start documenting how things were supposed to work. One of the first line items in that document: ALL processes and procedures will be documented prior to, or at the time of implementation. While that was not an issue int he department I managed, it was a sore sticking point for the only other GIS staff in Planning & Zoning, who steadfastly refused to document anything. There was not much that could be done about that, since that position was officially in another office. All that I could do was continue to emphasize the value of doing so. It never stuck, throughout the life of the project, so I just did it.

Background - Restoring a County GIS (Part 2)

I accepted the position of GIS Manager, just as I began my final year working towards a Bachelor of Science in Physical Geography (special emphasis on GIS and karst environments, along with a Minor in History). Little did I know that everything I had learned or been a part of in the past 20 years was about to be put to use.

The position reported directly to the County Assessor, a rural Republican woman from the north end of the County. She had held the seat for about 6 years I think, after being appointed from within the Office by the Governor, following the resignation of the previous Assessor. She was hard-nosed and demanding, while at the same time perfectly clear and reasonable in her expectations. Much like me, she preferred getting down to business with straight and honest answers to questions, and had very little tolerance for folks that beat around the bush. As a result, we got along really well, after testing each other a few times, and I was determined to make good things happen for her, the Office, and the County GIS.

The previous GIS Manager had left the position about 6 months prior to my arrival and only about a year in the position. I later discovered that this was a scenario that had repeated itself numerous times in the past. GIS folks signed on to the position long enough to hold the title, then left for a position in the private sector. There had been no attempt to retain anyone in the position, through commensurate compensation, and there was no interest in doing so. Consequently, the GIS was in a crumbling and dilapidated state, and not really much use to anyone except the persons maintaining and analyzing the records.

A GIS Analyst had only recently signed on, after having been absent from the profession for several years. She was doing her best maintain the data and day-to-day business; that is, in spite of a total lack of guidance, direction, documentation, or procedure, from the previous GIS Manager. I was concerned she would follow the same path as many others that had been in that position, but was fortunate to have her give the situation a chance and stay on through 4 years of the restructuring.

Throughout the County, there were only a couple of others involved with GIS. The GIS Coordinator within the Planning & Zoning Department mostly worked independently on projects associated with his department. Although he really had no responsibility to our office, he enthusiastically joined in most of the projects we rolled out during the next 5 years. The Director of Information Services was an immense help in forwarding our objectives too. He had originally been brought on years before, to initiate a GIS for the County. He had not worked in GIS for quite a few years, but understood the basic concepts; as well as the importance of a structured and well-documented approach to building and maintaining a useful GIS.

While this situation probably would have been quite a challenge for someone just finishing college, I had enough background to make a good run of it. I had already been in the business world for 20 years and just then leaving an Analyst role I had been in for 2 years with a financial software firm. Prior to that, I spent 5 years working with a team in my own company providing creative services, primarily in photography and event production, and pre-social media web marketing and promotional support. For the 13 years before that time, I proved my technical worth, rising through the ranks of a semi-local financial firm in various technology roles from quality assurance, helping structure production standards, statistical analytics, database management, data modeling, software and hardware support, software design and development, and even a little web production in the early days of HTML.

Initial Thoughts - Restoring a County GIS (Part 1)

During the past 5 years, I spent most of my waking hours rebuilding and restructuring a dilapidated Geographic Information System (GIS) for a Platte County in Missouri.

My team and I restructured systems and data to be more efficient, reliable, and accurate, while fully documenting processes and procedures, and organizing policy towards a functioning business solution using the ESRI suite of products, ArcGIS. Indeed, we only recently positioned our GIS to enable the Assessor and the County to begin to reduce its reliance on taxation and begin generating revenue through a fee structured subscription environment, capable of delivering geographic and non-geographic data to any client in the world, 24 hours a day, 365 days per year.

Resources for GIS were extremely limited, so it was very hands on, and I did most of the work on this system, personally. Along with a GIS Analyst under my supervision, the GIS Coordinator n the Planning & Zoning Department, as well as a bit of consulting on system design, architecture, and setup support from the Information Services Department.

In spite of numerous obstacles, both internal and external, we successfully upgraded all ArcGIS software from version 9.1 through v10.2, and implemented ArcGIS for Server v10.2, while at the same time, migrating this situation from an aging IBM DB2 v8.1 database on a Windows 2003 server to a Microsoft SQL Server 2008 R2 database on a Windows 2008 server.

This multi-part series will document those adventures in full, from beginning to the bitter end, when a newly-elected Republican Assessor, closely affiliated with the Tea Party, made one of several uninformed decisions. In his opinion, an [underpaid] GIS Manager was an unnecessary expense, because “nobody really uses maps anymore;” ironically, just a few days later he realized he had made a mistake. The GIS Manager did not just make maps.

Fortunately for him, the system was well documented and permitted the GIS Analyst to easily step in and start with a good foundation. That was one of project goals, after all; that anyone could easily step in and assume command of the GIS, with little effort. 

Essentially, this is a “Post Mortem” on the PCAO Geographic Information System Project, in every sense of the word, in the hopes of providing some insight to any other GIS Manager or Project Team that may face similar circumstances.