Category: Notes

GeoServer + PostgreSQL + OpenStreetMap

To load an openstreetmap datasource in PostgreSQL/PostGIS use the following command.

# osm2pgsql –slim –username XXX –password –database streetmap ./zuid-holland-latest.osm.pbf

osm2pgsql version 0.96.0 (64 bit id space)

Password:
Allocating memory for sparse node cache
Node-cache: cache=800MB, maxblocks=12800*65536, allocation method=9
Mid: pgsql, cache=800
Setting up table: planet_osm_nodes
Setting up table: planet_osm_ways
Setting up table: planet_osm_rels
Using built-in tag processing pipeline
Using projection SRS 3857 (Spherical Mercator)
Setting up table: planet_osm_point
Setting up table: planet_osm_line
Setting up table: planet_osm_polygon
Setting up table: planet_osm_roads

Reading in file: ./zuid-holland-latest.osm.pbf
Using PBF parser.
Processing: Node(15439k 270.9k/s) Way(2349k 71.20k/s) Relation(17020 1547.27/s) parse time: 101s
Node stats: total(15439302), max(6336341363) in 57s
Way stats: total(2349653), max(676644693) in 33s
Relation stats: total(17679), max(9386429) in 11s
Sorting data and creating indexes for planet_osm_point
Stopping table: planet_osm_nodes
Sorting data and creating indexes for planet_osm_roads
Sorting data and creating indexes for planet_osm_polygon
Stopping table: planet_osm_ways
Sorting data and creating indexes for planet_osm_line
Stopping table: planet_osm_rels
Stopped table: planet_osm_nodes in 0s
Building index on table: planet_osm_ways
Building index on table: planet_osm_rels
Copying planet_osm_roads to cluster by geometry finished
Creating geometry index on planet_osm_roads
Stopped table: planet_osm_rels in 2s
Creating osm_id index on planet_osm_roads
Creating indexes on planet_osm_roads finished
All indexes on planet_osm_roads created in 3s
Completed planet_osm_roads
Copying planet_osm_line to cluster by geometry finished
Creating geometry index on planet_osm_line
Copying planet_osm_point to cluster by geometry finished
Creating geometry index on planet_osm_point
Creating osm_id index on planet_osm_line
Creating indexes on planet_osm_line finished
All indexes on planet_osm_line created in 22s
Completed planet_osm_line
Creating osm_id index on planet_osm_point
Creating indexes on planet_osm_point finished
All indexes on planet_osm_point created in 65s
Completed planet_osm_point
Stopped table: planet_osm_ways in 98s
Copying planet_osm_polygon to cluster by geometry finished
Creating geometry index on planet_osm_polygon
Creating osm_id index on planet_osm_polygon
Creating indexes on planet_osm_polygon finished
All indexes on planet_osm_polygon created in 252s
Completed planet_osm_polygon

Osm2pgsql took 355s overall
node cache: stored: 15439302(100.00%), storage efficiency: 50.00% (dense blocks: 0, sparse nodes: 15439302), hit rate: 100.00%

Create GIS database in PostgreSQL

1. Install PostgreSQL database (for example 9.6)

2. Install postgis module (for example 2.5.1)

3. Start pgAdmin 4

4. Create database

5. Enable GIS features in database with following command:
CREATE EXTENSION postgis;

6. Now shapefile can be loaded with PostGIS 2.0 shapefile tool

7. Set GEO input format (for example Rijksdriehoek)
SELECT UpdateGeometrySRID(‘wegvakken’,’geom’, 28992);

8. Create WGS84 geog column and fill it.
ALTER TABLE wegvakken ADD COLUMN geog geography(geometry);
UPDATE wegvakken SET geog = ST_Transform(geom, 4326);
CREATE INDEX wegvakken_geog_idx ON wegvakken USING GIST (geog);

9. Now pgAdmin4 can show GEO layer with OpenStreetMap layer as base.

10. Set center point of RijksDriehoek (Onze Lieve Vrouwetoren (‘Lange Jan’) in Amersfoort.)
SELECT * FROM wegvakken ORDER BY geom <-> ‘SRID=28992;POINT(155000 463000)’::geometry LIMIT 1;

11. select ST_X(geom), ST_Y(geom) from shapefile

Windows 10 upgrade takes hours

This afternoon my Quad Core 3GHZ Desktop finally received the Windows 10 creator patch. It took my desktop more than 4 hours to install it. Why is a Windows 10 patch always so slow? If i upgrade a Raspberry Pi Quad Core 1.2GHZ Linux kernel it’s only take 10 minutes. Strange that low end linux hardware upgrades are must faster than high end Windows hardware. I think that the Microsoft staff must go on training to learn how to patch Operating Systems fast and without reboots 🙂

Raspberry Pi Noobs cleanup

If you want to have a nice and clean Raspberry Pi Noobs installation. Execute te followings commands to remove unless tools:

sudo apt-get purge wolfram-engine (Reduce image size with 683MB)
sudo apt-get purge sonic-pi (Reduce image size with 151MB)
sudo apt-get purge scratch (Reduce image size with 93MB)
sudo apt-get purge scratch2 (Reduce image size with 147MB)
sudo apt-get autoclean

This initial Noobs image size will now be arround 3GB 🙂

Google Charts zoom in dataset example

google-chartsI took me a while to understand how Google Charts can be extended with a zoom/drill in data set feature. You have to add then google.visualization.events.addListener method which fetch the selected datapoint of a graph. With some extra javascript code you can reload the chart with a more detailed dataset. See example below!

<script type="text/javascript" src="https://www.google.com/jsapi"></script>
    <script type="text/javascript">
      google.load("visualization", "1", {packages:["bar"]});
      google.setOnLoadCallback(drawChart);
      function drawChart() {

      var options = {
          bars: "vertical",
          bar: {groupWidth: "90%"},
          legend: { position: "none"},
          vAxis: {format: "decimal"},
          isStacked:true,colors: ["#0066cc", "#808080"],vAxis: { format:"decimal", viewWindow: { min: 0, max: ".round($max_forecast+100)." } }, series: {
            3: {
                targetAxisIndex: 1
            }
          }
        };

        var data = google.visualization.arrayToDataTable([['','Low (kWh)','Normal (kWh)','Solar (kWh)','Forecast (kWh)'],['2006',0,0,0,0],['2007',0,0,0,0],['2008',0,0,0,0],['2009',0,0,0,0],['2010',0,0,0,0],['2011',0,0,0,0],['2012',0,0,0,0],['2013',0,0,0,0],['2014',0,0,0,0],['2015',0,0,0,0],['2016',20,20,11,278.75]]);
        var chart = new google.charts.Bar(document.getElementById("chart_div"));
        chart.draw(data, google.charts.Bar.convertOptions(options));

        google.visualization.events.addListener(chart, "select", selectHandler);

        function selectHandler(e)     {
           var year = data.getValue(chart.getSelection()[0].row, 0);
           link("pid=30&eid=88&date="+year+"-1-1");
        }
      }

function link(value)
{
	var form = document.forms['plaatenergy'];
	var newInput = document.createElement('input');
	newInput.setAttribute('type','hidden');
	newInput.setAttribute('name','token');
	newInput.setAttribute('value',value);
	form.appendChild(newInput);		
		
	form.submit();
}

Oracle SQL Loader

control file dwh.ctl

LOAD DATA
APPEND INTO TABLE CDR_UNRATED
(
WK_BMP_USAGE INTEGER EXTERNAL TERMINATED BY “,”,
WK_USAGE_TYPE INTEGER EXTERNAL TERMINATED BY “,”,
WK_EXCHANGE_ID INTEGER EXTERNAL TERMINATED BY “,”,
WK_TRANS_DT INTEGER EXTERNAL TERMINATED BY “,”,
WK_TRANS_TM INTEGER EXTERNAL TERMINATED BY “,”,
CDR_SEQUENCE_NUMBER INTEGER EXTERNAL TERMINATED BY “,”,
TRANS_DT DATE “ddmmyyyyhh24miss” TERMINATED BY “,”,
UNITS INTEGER EXTERNAL TERMINATED BY “,”,
RELATED_CALL_NUMBER INTEGER EXTERNAL TERMINATED BY “,”,
A_NUMBER CHAR TERMINATED BY “,”,
C_NUMBER CHAR TERMINATED BY “,”,
IN_SERVICE_IND CHAR TERMINATED BY “,”,
CALL_STATUS CHAR TERMINATED BY “,”,
MARKETING_CALL_STATUS CHAR TERMINATED BY “,”,
ANSWER_TIME INTEGER EXTERNAL TERMINATED BY “,”,
B_NUMBER CHAR TERMINATED BY “,”,
CHARGED_NUMBER CHAR TERMINATED BY “,”,
EXTERNAL_TYPE_ID INTEGER EXTERNAL TERMINATED BY “,”,
B_CATEGORY CHAR TERMINATED BY “,”,
SERVICE_TERMINATING CHAR TERMINATED BY “,”,
EXCEPTION_CODE CHAR TERMINATED BY “,”,
MEDIA_NUMBER CHAR TERMINATED BY “,”
)

CSV file
086,01,12345678912345,12345678912345,31122006,235959,1,31122006235959,12,0301234567,0301234567,10,U,A,5,182123451,282123451,1,15,I,abcdeabcdeabcde,1

Start SQL Loader
$ sqlldr TCT/TCT@ORCL control=dwh.ctl log=dwh_import.log data=DHW_20060501235959.csv

Oracle Application Server commands

Oracle Application Server

Show Status
$ opmnctl status

Show Cluster status
$ opmnctl @cluster status

Stop single process
$ opmnctl stopproc -process-type=XXX

Start single process
$ opmnctl startproc -process-type=XXX

Stop all processes for a cluster
$ opmnctl @cluster stopall

Start all processes for a cluster
$ opmnctl @cluster startall

———————————————

Create new instance
$ createinstance – instanceName XXXX
$ opmnctl reload
$ opmnctl startall
Note: New instance will create a new group!

Remove instance
$ removeinstance -instanceName XXXX

———————————————-

Cluster IP address = 225.0.0.2
Cluster Port Number = 6790

———————————————-

Solaris ok prompt

# OK prompt
How to get to the OK Prompt You can shutdown the system using init 0
Reboot the system and press the STOP + A keys together as the boot time messages appear on the screen
Sets diagnostic mode ( diag-switch? True ) Pressing STOP and D Keys
Sets configuration variables to default value Pressing STOP and N Keys

# Commands
OK showdevs
OK show-disks
OK boot /dev/dsk/c0t0d0s0 -s
OK boot cdrom -s

# Boot Single User Mode:
OK> boot -s

Solaris create ALOM network

sc> shownetwork
SC network configuration is:
IP Address: 0.0.0.0
Gateway address: 0.0.0.0
Netmask: 255.255.255.0
Ethernet address: 00:03:ba:57:76:8e
sc> setupsc
Entering Interactive setup mode. To exit and discard changes to that point, use Ctrl-C or to exit and save changes to that point, use Ctrl-Z.
Do you wish to configure the enabled interfaces [y]? y
Should the SC network interface be enabled [y]? y
Should the SC email alerts be enabled [y]? n
Do you wish to configure the network interface [y]? y
Should the SC use DHCP to obtain its network configuration [n]? n
Enter the SC IP address [100.100.100.100]? 192.168.0.225
Enter the SC IP netmask [255.255.255.0]? 255.255.255.240
Enter the SC IP gateway address [100.100.100.100]? 192.168.0.228
Should the TPE link test be enabled [y]? y
Do you wish to configure the managed system interface [y]? n
Do you wish to configure the network management interfaces [y]? n
Do you wish to configure the SC parameters [y]? n
Your ALOM configuration profile has been successfully completed. To activate your network configuration, please reset the SC.
sc> resetsc
Are you sure you want to reset the SC [y/n]? y
User Requested SC Shutdown

SC Alert: SC System booted.

SC Alert: AC power unavailable for PSU @ PS0.

[SC Status 7fff – OK] Enter #. to return to ALOM.
Oct 18 14:16:57 nlnwhabcm03 rmclomv: SC unretrieved msg OCT 18 13:16:54 2004 UTC [SC System booted.]

Copyright 2003 Sun Microsystems, Inc. All rights reserved.
Use is subject to license terms.

Sun(tm) Advanced Lights Out Manager 1.3 (nlnwhabcm03)

Please login: admin
Please Enter password: ******

sc> shownetwork
SC network configuration is:
IP Address: 192.168.0.225
Gateway address: 192.168.0.228
Netmask: 255.255.255.240
Ethernet address: 00:03:ba:57:76:8e
sc>

# Open console by ALOM port

$ telnet db01-alom
– login: admin
– password: admin

$ console -f
– login: root
– password: <>

Now your are login!

Solaris cluster commands

hastatus (Show cluster status)
hares -offline -sys (Stop component)
hares -online -sys (Start component)

hagrp -clear (Clear groep die gecrashed is)

hastatus -sum # Show autodisable groups
hagrp -autoenable

vxprint – Show status Veritas

# Check Veritas Cluster main.cf file
hacf -verify /etc/VRTSvcs/conf/config -display

# Configuration Read /Write
haconf -makerw
hares -modify DB_OP_XX_SID Critical 0
hares -value DB_OP_XX_SID Critical
hares -modify DB_OP_XX_SID Enabled 0
hares -value DB_OP_XX_SID Enabled
haconf -dump -makero

SUN StorEdge Backup Solution scripts

# General
/usr/sbin/nsr/nsrim -v (Print Volume information of all used tapes)

# ——————————————————————————
# BACKUP (SINGLE TAPE DEVICE)
# ——————————————————————————

0 19 * * 1-5 /usr/sbin/nsr/tape_label_mount.sh >> /nsr/logs/tape.log
#!/bin/sh
#
# Load / relabel / mount tape
#
/usr/sbin/nsr/nsrmm -l -R << EOF y EOF sleep 30 /usr/sbin/nsr/nsrmm -m 20:30 start total backup StorEdge backup software regeld dit! 0 6 * * 1-5 /usr/sbin/nsr/tape_unmount.sh >> /nsr/logs/tape.log
#!/bin/sh
#
# unLoad /eject tape
#
/usr/sbin/nsr/nsrmm -j

# ——————————————————————————
# BACKUP (TAPE ROBOT 8 POSITIES)
# ——————————————————————————

00:30 start load/relabel/mount script

#!/bin/sh
#
# Load / label / mount tape
#
# Volume Name Tape Location Note
# volume.001 Location 1 Monday
# volume.002 Location 2 Tuesday
# volume.003 Location 3 Wedneyday
# volume.004 Location 4 Thursday
# volume.005 Location 5 Friday
# volume.006 Location 6 Saterday
# volume.007 Location 7 Sunday
# Location 8 Clean tape location

DAYOFWEEK=`date +”%u`

# Load / relabel tape (-Y No approve of action is needed)
/usr/sbin/nsr/nsrjb -Y -L -S ${DAYOFWEEK} svolume.00${DAYOFWEEK}

# Mount tape
/usr/sbin/nsr/nsrjb -l -S ${DAYOFWEEK}

01:00 start total backup
StorEdge backup software regeld dit!

23:30 start unmount / unmount script

#!/bin/sh
#
# unLoad tape
#

DAYOFWEEK=`date +”%u`

# unmount / unload tape
/usr/sbin/nsr/nsrjb -u -S ${DAYOFWEEK}

Veritas Commands

Veritas

Producten:
– Veritas File System Manager
– Veritas Volume manager
– Veritas Cluster

Commandos:
vxdg list : Show Disk Silyses

Veritas Volume Manager:
– Start GUI
# export DISPLAY=172.16.1.177:0.0
# vea &
Login=root Password=Default
– Stop
# vxdctl stop
# vxiod -f set 0

Veritas Volume Cluster:
– Start GUI
# export DISPLAY=172.16.1.177:0.0
# hagui &
Login=admin Password=Password

Change Parity:
# format
% select disk
% parity
% 7 (select parity 7)
% size = 0c
% label

Update License:
# mkdir /etc/vx/licenses/lic/old
# mv /etc/vx/licenses/lic/* /etc/vx/licenses/lic/old ; /opt/VRTSvlic/bin/vxlicinst
bv. 3ZPV-NCCT-H7MH-VL6T-Z36X-RRC

# Check cluser config
hacf -verify /etc/VRTSvcs/conf/config

MySQL commands

########
LOAD DUMP

After you have dumped out your data into a file as described here, FTP or scp that dump file to the home directory (/) on our system.

Once you have uploaded the dump file to your account here, get a shell prompt on our system using telnet or ssh.

Now import the dump file into MySQL by typing all the following on 1 single line at the shell prompt:

mysql -p -h DBSERVER dbname < dbname.sql The above assumes that your database name on our system is "dbname" and the dumpfile that you uploaded was named "dbname.sql". Replace those with your correct database name and dumpfile filename. Also replace DBSERVER with your correct database server name. ####### Creating MySQL database on Linux system 1. I assume that you are working from your account and not the root. Start a terminal session and become the superuser (Type su at the prompt and then enter the root password). 2. Now we'll access the MySQL server. Type: mysql -u root -p The system prompts for the MySQL root password that you set up in Installing MySQL on Linux. (Note: This is not the Linux root password but the MySQL root password). Enter the password, which is not displayed for security reasons. Once you are successfully logged in, the system prints a welcome message and displays the mysql prompt ... something like Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1 to server version: 3.22.32 Type 'help' for help. mysql>

3. Now we are ready for creating the employees database. Issue the command:

create database employees;

(Note: The command ends with a semi-colon)

4. An important point to note is that this database is created by the root and so will not be accessible to any other user unless permitted by the root. Thus, in order to use this database from my account (called manish), I have to set the permissions by issuing the following command:

GRANT ALL ON employees.* TO manish@localhost IDENTIFIED BY “eagle”

The above command grants my account (manish@localhost) all the permissions on employees database and sets my password to eagle. You should replace manish with your user name and choose an appropriate password.
5. Close the mysql session by typing quit at the prompt. Exit from superuser and come back to your account. (Type exit).
6. To connect to MySQL from your account, type:

mysql -u user_name -p

Type in the password when prompted. (This password was set by the GRANTS ALL… command above) . The system displays the welcome message once you have successfully logged on to MySQL. Here is how your session should look like:

[manish@localhost manish]$ mysql -u manish -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 3 to server version: 3.22.32

Type ‘help’ for help.

mysql>

7. Typing the command SHOW DATABASES; will list all the databases available on the system. You should get a display similar to:

mysql> SHOW DATABASES;
+—————-+
| Database |
+—————-+
| employees |
| mysql |
| test |
+—————-+
3 rows in set (0.00 sec)

8. Enter quit at the mysql> prompt to come out of the mysql client program.

######
SET OLD PASSWORD

Reset the password to pre-4.1 style for each user that needs to use a pre-4.1 client program. This can be done using the SET PASSWORD statement and the OLD_PASSWORD() function:

mysql> SET PASSWORD FOR ‘some_user’@’some_host’ = OLD_PASSWORD(‘newpwd’);
mysql> FLUSH PRIVILEGES;

Substitute the password you want to use for “newpwd” in the preceding examples. MySQL cannot tell you what the original password was, so you’ll need to pick a new one.

######
Drop database

Definition: The drop database command is used when you no longer need one of the SQL databases on your server. It will remove it permanently. It is phrased as: drop database [DatabaseName];
Examples: This will remove the database ‘Dresses’ from the MySQL server:

mysql> drop database Dresses;