[ Ø ] Harsh Prakash – GIS Blog

Quiet Musings On Applied Spatial (Health, Disaster, Technology, Planning et al.)

Archive for the ‘Programming’ Category

Esri in AWS Cloud

without comments

Background, Options, Opportunities

Esri in AWS Cloud

Written by Harsh

December 17th, 2016 at 11:48 pm

Posted in GIS,Management,Programming,Technology

Tagged with , ,

HowTo: Run ‘ArcGIS for Server Advanced Enterprise’ (10.3.1) on Amazon EC2 Red Hat Enterprise Linux (7)

without comments

The talks on ArcGIS Server at ESRI Health GIS were fun, but I wanted more – specifically, to install and administer its latest release on Amazon Web Services, all via the trusted command line. Here’s how I did that:

To follow along, get an EDN license and an AWS account. Especially, if you have been in the industry for long, there’s no good excuse to not have those with the biggest companies in GIS and da Cloud (and while you are at it, get MapBox and CartoDB accounts too).


### Setup the stage ###
# Downloaded its AWS key from //aws.amazon.com/console/ and connected to my instance (ensured it matched the min. system requirements) using its public DNS (if you restart your instance, this will change). Note I SSHed using Cygwin instead of PuTTy.
$ ssh -i "key.pem" ec2-user@#.#.#.#.compute.amazonaws.com
$ cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.1 (Maipo) # Even though I used RHEL-7.0_HVM_GA-20141017-x86_64-1-Hourly2-GP2 by Red Hat (I later found out that ESRI provides its own AMI)
$ sudo yum upgrade
$ sudo yum update
$ sudo yum install emacs # For that college-dorm smell, no offense Nano/Vi
$ sudo emacs ~/.bashrc
    force_color_prompt=yes # If you haven't already... (Ignored the embedded rant and uncommented this line to make the prompt colored so it was easier to read in-between)

### Setup the instance ###
# I used a M4.LARGE instance with a 20GB EBS volume (in the same Availability Zone, of course) - ensured it didn't go away if I were to terminate the instance. Then, I extended the partition to exceed the min. space requirements (took a snapshot first) - unfortunately, AWS docs didn't help much with that.
$ df -h
> ...
$ lsblk # Listed block partitions attached to the device. Since there was a gap in sizes between the partition and the device (and there were no other partitions), I resized the child partition "XVDA2" (the root file system where I would finally install ArcGIS Server) to use up the surplus space on its parent disk "XVDA".
> NAME SIZE TYPE MOUNTPOINT
> xvda 20G disk
> |_xvda2 6G part /
# First, updated its metadata in the partition table
$ sudo yum install gdisk # Since disk label was GPT
$ sudo gdisk /dev/xvda/
$     print # Noted the start sector
$     delete
$     new
$     #### # Used the same start sector so that data is preserved
$     \r # For the max. last sector
$     # # Used the same partition code
$     print
$     write
$     y
# Next, updated the actual XFS file system
$ sudo xfs_growfs / # This is the actual change for XFS. If 'df -T' reveals the older EXT4, use 'resize2fs'.
# Then, confirmed to see if the boot sector was present so that stop-start will work
$ sudo file -s /dev/xvda # Bootloader
# Finally, rebooted the instance to reflect the new size
$ sudo reboot

### Onto GIStuff ###
# WinSCPed and untarred the fresh-off-the-press 1GB release
$ tar -xvf ArcGIS_for_server_linux_1031_145870.gz
# Got the right ECP#########?
$ ./Setup # Started headless installation - try "--verbose" if you run into other issues
# Hit a diagnostics roadblock: File handle limits for the install user were required to be set to 65535 and the number of processes limits to 25059. So...
$ sudo emacs /etc/security/limits.conf
$     ec2-user soft nofile 65535
$     ec2-user hard nofile 65535
$     ec2-user soft nproc 25059
$     ec2-user hard nproc 25059
# Logged out, logged back in, verified
$ ulimit -Hn -Hu
$ ulimit -Sn -Su
$ ./Setup

### Authorize, authorize, authorize! ###
# Created and uploaded authorization.txt, and downloaded authorization.ecp from //my.esri.com/ -> "My Organization" -> "Licensing" -> "Secure Site Operations"
$ locate -i authorization.ecp
$ readlink -f authorization.ecp
$ ./authorizeSoftware -f /path/authorization.ecp
$ ./authorizeSoftware -s # s=status, not silent
$ ./startserver.sh
$ netstat -lnp | grep "6080" # Confirmed owned processes - that it was listening on the default TCP@6080 (port is only required if you don't have the Web Adapter)
# Ensured IP and domain were listed correctly in the hosts file (e.g. Single IP may be mapped to multiple hosts, both IPv4 and IPv6 may be mapped to a single host, etc.)
$ hostname
$ emacs /etc/hosts
$     127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
$     ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
$     #.#.#.# localhost localhost.localdomain localhost4 localhost4.localdomain4
# But wait, before I could browse to my site from a public browser, I needed to add this Inbound Rule to the Security Group attached to the instance
Custom TCP rule TCP 6080 0.0.0.0/0

### Browser ahoy! ###
//#.#.#.# or machinename:6080/arcgis/manager
ArcGIS Server Setup Wizard -> Create New Site
Primary Site Administrator -> Create Account # Stored with the site, not the OS
# Must be local and accessible from every machine in your site
    Root Server Directory: /home/username/arcgis/server/usr/directories # To store output images, etc.
    Configuration Store: /home/username/arcgis/server/usr/config-store # To hold info about the server's machines, services, directories, etc.
# This is when I ran into "0x80040154 - Could not create object 'ConfigurationFactory'". So, went digging through the logs...
$ cat /home/ec2-user/arcgis/server/usr/logs/EC2/server/server-...log
> ...
> Cluster 'default' successfully created.
> Failed to create the site. com.esri.arcgis.discovery.servicelib.AGSException: java.lang.Exception: AutomationException: 0x80040154 - Could not create object 'ConfigurationFactory'.
> Disconnecting the site from the configuration store.
...
# Back to the server: File/directory permission issue? Nope. The issue turned out to be missing packages, even though the pre-installation dependencies check had passed. All 15 listed below:
$ sudo yum list installed
$ sudo yum install wget
$ wget http://vault.centos.org/6.2/os/x86_64/Packages/xorg-x11-server-Xvfb-1.10.4-6.el6.x86_64.rpm
$ sudo yum localinstall xorg-x11-server-Xvfb-1.10.4-6.el6.x86_64.rpm
sudo yum install Xvfb # Else "Unable to start Xvfb on any port in the range 6600-6619"
sudo yum install freetype
sudo yum install fontconfig
sudo yum install mesa-libGL
sudo yum install mesa-libGLU
sudo yum install redhat-lsb
sudo yum install glibc
sudo yum install libXtst
sudo yum install libXext
sudo yum install libX11
sudo yum install libXi
sudo yum install libXdmcp
sudo yum install libXrender
sudo yum install libXau
# Cleanliness is next to godliness, or so my Catholic school nuns would say
$ sudo yum clean all
$ cd /tmp/
$ sudo rm -r *
$ logout

### Back to the browser ###
//#.#.#.#:6080/arcgis/manager/
# At the end, added SSL using a self-signed certificate
//#.#.#.#:6080/arcgis/admin/
Custom TCP rule TCP 6443 0.0.0.0/0 # Added this rule to the group on AWS first

### Uninstall? ###
$ ./stopserver.sh
$ ./uninstall_ArcGISServer
# rm folders after done

Conclusion: 6443 or 8443?

After years of doing this with first ESRI (PROD), then MapServer (PROD) and GeoServer (DEV), I went back to the dark ahem ESRI side. And what do I keep finding? That the big two are blending together in terms of looks. E.g. The console of the other Java-powered mapping server, GeoServer, is looking similar to that of its big brother on-steroids. The third, MapServer, somewhat paradoxically on the other hand, has both come a long way (MapCache and ScribeUI, yay!) and still lost ground.

Next up, testing Tippecanoe.

PS:
* I tried both 10.3.1 and 10.0 on Ubuntu (15.04), unsupported. While both installed, site creation didn’t work because of missing packages – searching through apt-cache didn’t help either. On Windows, there is always their CloudBuilder.

Related:
* GeoNet
* Landsat on AWS in ArcGIS

Written by Harsh

September 28th, 2015 at 7:43 pm

Posted in GIS,IMS,Programming

Tagged with ,

How We Balanced Proprietary With Opensource Software And Saved Tax Dollars, And You Can Too

without comments

It all began with a question – “Can we do with out?”.

GIS@NIH

Enterprise Architecture > Technology Architecture > Geographic Information System (GIS):
* Geographic Information System (GIS) Pattern
* GIS Desktop Brick
* GIS Virtual Globe Brick
* GIS IMS Brick
* GIS Web Service Brick

Related:
* GIS Market Study of Internet Mapping Server (IMS) – Summary – Requirements and Comparison Matrix (2006)

Monty Hall: Stay or Switch Your Pick?

without comments

Written by Harsh

January 31st, 2013 at 10:48 pm

A Touch of Play

with 3 comments

First impressions after testing Microsoft’s Surface Table:

Pi: Quite MusingIt is a coffee-table sized hardware running Windows Vista and allowing collaborative interaction from up to 4-6 participants. The number of hand-gestures it can recognize is obviously higher than that of a standard touch-screen which can typically handle only a single tap and drag, and maybe multi-touch. On the other hand, the Surface Table can recognize multiple taps, imprecise flicks and resizes, and touch-intensity. Actually, much like a TouchSmart, it can even detect movement just above its surface. Simply put, it is like a giant iPhone.

Application

So how does it lend itself to GIS/Planning application development? Well, it is more eye-candy than useful for its cost at this point and appropriate application ideas may not come readily. If you try to recreate a similar collaborative environment with a series of Tablet PCs, TouchSmarts and Windows 7, you might just be successful. Note that it can’t be detached from its base and wall-mounted since it has a projector underneath.

The Surface Table’s biggest strength lies in its enabling a collaborative environment, and therefore, it is more suited towards “playful infotainment”-type applications. If you develop GIS/Planning applications for the Surface Table, note this: It would be a lot of fun, but maybe not a lot useful. And also, it doesn’t carry any browser application (!) so you can’t simply start using your planning mash-up and development would present its own WPF learning curve for the web savvy. For an elegant GUI design, remember that fat shaky fingers need big buttons. In terms of pricing, Microsoft is currently also charging for its SDK (approx. $3K): Not sure of their pricing model, but it doesn’t seem like a smart idea if their goal is to also encourage the Viral Phenomenon. And although, they don’t yet come pre-installed (!), a wireless card and wheels can easily be mounted to turn your Surface Table into a self-contained unit to enhance its portability.

Sync

There are already some creative applications in-use: Soldiers returning from a patrol dump their head gears onto the Surface Table, and its docking corner instantly syncs their captured data with their sync folder- no fumbling there! Special ID tags can “identify” themselves to the Surface Table, but cell phones running Windows Mobile require a download before they can sync. Selected Omni Sheraton hotels and others are currently showcasing Surface Tables.

Technology

So how does it work? Well, conventional technologies detect touch-location by interrupting:
* Infrared
* Optical Field
* Surface Acoustic Wave
This interception happens just above the screen substrata and its grid coordinates are then translated to screen position. Alternatively, you can do a makeover of your current display using Dispersive Signal Technology (DST). DST integrates chemically-strengthened glass onto existing display. It detects bending wave within the glass radiating to the 4 corners where it gets converted to electric signals. This approach also makes it ideal for heavy-duty use to filter out “noise”, say when outdoors or think glass spills and crumbs in a snack-rich community planning meeting. Then there is Proximity Capacitive Resistance (PCR) for touch-across-surface.

Written by Harsh

January 20th, 2009 at 4:54 am

b2evolution 2 wordpress

with 3 comments

Well, I have switched from b2evolution to WordPress CMS. And thanks to Apache’s mod_rewrite, I was able to keep all my old links intact. Here’s how:

### wordpress:
<IfModule mod_rewrite.c>
# basic:
RewriteEngine On
RewriteBase /gistools/discuss/weblogs/blogs/
# file:
RewriteCond %{REQUEST_FILENAME} !-f
# dir:
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /gistools/discuss/weblogs/blogs/index.php [L]
# [R] Redirect [L] Last rule
# post:
Options +FollowSymLinks
RewriteCond %{QUERY_STRING} title=([^&]+)
RewriteRule ^index\.php /gistools/discuss/weblogs/blogs/%1\.html? [r=301,nc]
# archive – monthly:
http://www.spatiallink.org/gistools/discuss/weblogs/blogs/pi.php?m=200807
RewriteCond %{QUERY_STRING} m=([0-9]{4})([0-9]{2})
RewriteRule ^index\.php /gistools/discuss/weblogs/blogs/%1/%2? [r=301,nc]
# archive – category:
RewriteCond %{QUERY_STRING} cat=15
RewriteRule ^index\.php /blog/category/aspatial [r=301,nc]
RewriteCond %{QUERY_STRING} cat=14
RewriteRule ^index\.php /blog/category/spatial [r=301,nc]
</IfModule>
### end wordpress

This was how the old URL looked like, http://www.spatiallink.org/gistools/discuss/weblogs/blogs/?title=gisp_and_aicp. Note that there were limitations to permalink, since %year%, %day% or %category% were unknown from the old URL. Fortunately, I had only 2 categories, so this was a cinch.

Written by Harsh

July 28th, 2008 at 11:22 pm

Why Contribute

without comments

Paul Ramsey points to Danny de Vries‘s take on Free and Open Source Software for Geospatial [FOSS4G] 2007:

“What we saw was a young and passionate movement not-so-subtly showcasing their dedication for open-source as a tool by which to challenge corporate, or closed-source, IT monopolies in the geospatial domain.”

I want to underline the ‘showcasing’ part. It is important to not ignore why that is significant for contribution to opensource, which as some would like you to believe is often lacking direction and profit and not the best use of your time. And it can be summarized like so:

                        +—[IN]—> LEARN
CONTRIBUTE —|
                        +—[OUT]—> SHOWCASE —> GET WORK

–π

Related:
• My Pick of FOSS4G 2007 Presentation Submissions
• Contribute

Written by Harsh

November 22nd, 2007 at 12:07 am

Posted in OSGeo,Programming

Tagged with ,

A Tale of Two Languages

with 2 comments

Try this page to compare Ruby‘s and Python‘s language elegance side-by-side. Spoiler Warning: There is a winner!

To get you started:
Ruby – string.method [“String”.reverse or “String”.length]
Python – string[slice] or function(string) [“String”[::-1] or len(“String”)]

–π

Related:
• Python Interpreter
• ASP
• Cold Fusion
• JSP
• Perl [ActivePerl]
• [ActivePython]
• PHP
• Tcl [ActiveTcl]
• A Tale of Two Cities

Written by Harsh

November 11th, 2007 at 10:36 pm

Posted in Programming,Web

Tagged with ,

WIKI: Edit Lock Schema

with 3 comments

Now that I would update the DFIRM WIKI more frequently, I added a lock this past weekend to prevent simultaneous editing. And after being hit by abuse through automated comments, basic verification was also added while still allowing relatively hassle-free editing.
Pi: Quiet Musing
At some point, I may submit these improvements back to TipiWiki.

–π

Written by Harsh

June 17th, 2005 at 7:34 pm

Posted in Programming,Web

Tagged with , , ,

Half-life of a Webpage

without comments

The primary objective of this blog is to mull over industry trends and abstract ideas relevant to the profession, not to regurgitate “operational details”. However, this post may bend that rule.

For those not in the know, a webpage does a lot of behind-the-scene work before it spits-out text on the screen. Here’s a summary of what this webpage does:

• The very first thing it does is send out a header depending on the client-browser. This is recommended when, say, different protocols are used to access the webpage. Note that this step gets initiated only after the Apache Webserver has finished running through its configuration directives. The webpage then marks the start-time for script download and execution. Measuring script download and execution time helps in diagnostics. The webpage also goes down a list of red-flags checking for browser compatibility and permission-settings. Later, it establishes connections with MySQL databases and fetches or defines client and script variables.

• Only then does the layout begin to emerge with some CSS, XHTML and plenty of include files. Care is taken to separate presentation which has been kept to a minimum given the volunteer nature of the website, from content and function, and make it easier to reuse data. To display news feeds, as is the case here, the webpage fetches the feed URL and slices its content into nodes. Sometimes feed URLs do not provide information as desired. For example, this feed URL does not provide a direct hyperlink to its article. Sometimes a feed URL includes an image-path in its description that needs to be dropped. For such cases, scripting languages like PHP offer a wide-array of string-manipulation functions. It is advisable to ensure that the webpage continues to get parsed in a timely manner even if the fetching fails.

• The webpage then wraps-up logging of relevant variables and closes open database connections. If script execution has generated any errors, a summary gets emailed to the administrator. The webpage then spits-out the footer. Its decay into dead text is finally complete […well, unless you use AJAX to monitor client-behavior, as is the case here].

• A quick note on the website maintenance: Given its volunteer nature, it is maintained in small nudges i.e. “minor increments made frequently”, with the emphasis being on function over form.

Related:
• World Wide Web Consortium
• Web Style Guide
• Interesting Website
  º http://www.nyas.org/
  º http://news.google.com/
  º http://www.cancer.gov/
  º http://www.nobodyhere.com/
• Website Theme: A lot of experiences came together to start and shape the evolving theme of this website- During the 2002 Colorado/Arizona wildfire disaster, I received an email from the FGDC list serve requesting volunteers for assistance; Then at the 2003 ESRI Annual Conference, I learnt how volunteering is not easy- how the volunteer is not always in control; The omnipresence of mature opensource software not getting enough attention from the general public was a cause for concern; Also, a need was felt to enhance the functionality of my cellphone by connecting it with custom online applications; Additionally, there was a personal need to digest vast amounts of professional information from anywhere.

–π

Written by Harsh

May 8th, 2005 at 10:55 pm

Posted in Programming,Web

Tagged with ,