Uploading a Graph Database to the Web

Okay, you have a graph database you've created locally, and you now wish to share it on the web. How do you go about doing that? There are several approaches: you can use AWS' graph database, Neptune. You can use GraphGist from neo4j. This article will go with graphenedb.com's graph database.

Simply follow the instructions on their site (here), and you have a graph database uploaded for sharing and collaboration.

Share a Public-Acess/Read-only Clone of your Graph Database

Problem

So, you want to show off the amazing tech you've been developing to your potential buyers using Graph Database technology. How do you do it safely?

Well, the safest way is not to and you can show off power-point presentations to your heart's content.

But the power-point presentation, no matter how slick, is nothing to what your graph database can do for your customers, and you know it, and I know it. Believe me, I've been involve in generating twenty power point presentations, and the customers just don't get it until you put the graph database in their hands and let them play with it (that's what I do, and that's how we got the sale, and then the follow-on contract).

Solution

So, back to sharing your graph database safely. How do you go about doing that?

We use grapheneDB.com as our DaaS over neo4j graph database, and they provide tools that makes sharing a copy of your database safely a cinch. Let's see how.

Let's say you have this graph database in your private access that you wish to share:

You're not going to share your database (you could, but we won't in this walkthrough), instead, we'll make a clone of that database. Here's how.  As you see, you have an [upgrade/clone] option for your database, select that.

Clone Database

By selecting [upgrade/clone] you have two options: clone from running database that involves downtime (that's why I'm doing this on a Sunday, see), or clone database from a backup. 

As you've been making daily backups (link) – right? – you can select the clone from a backup option. This is available at the professional editions. As I have a developer database here, I'll be cloning directly from this currently running database.

That's the top-half of the page, the second half is choosing what the cloned database will look like, which will be usually be the same configuration as your source database. The one field you need to supply is the new database name. Since this is a public access database, I'm prefixing the new name with 'PUBLIC': a little reminder-tag for me.

Once you have the configuration you desire for the new database select [Create database]. You get a nice little warning. Select [Continue].

Then you get the spinny-box as it creates the new database...

la-di-dah! Then you have your new database! YAY!

Configure New/Cloned Database to Read-Only Mode

But you're not done yet! We wish to provide public access to our database, but we don't want our adoring public to be creating THREE BILLION NODES on their own and drive you out of business.

Not that they would do that, ever, but still.

So, let's make this public-facing clone safe for access by making it read-only. Go to the Configure-tab on your new PUBLIC database. There you see that the read-only mode is "DISABLED":

Let's change that. Select [Edit configuration], and then select the "Enable"-link under Read only mode. Once you do that, select [Apply and restart].

You get a warning-dialog, so, again, select the red [Apply and restart] in this dialog:

You again get a 'restarting database'-spinny:

Then a confirmation at the top of your browser that your database has been restarted with the new configuration:

So that now, when you select the Overview-tab, you are reinforced with the confirmation that your database-clone is now in read-only mode:

Confirmed Read-Only

Let's confirm this by trying to create a node in the neo4j browser. Select [Launch] in your Overview-tab. Since this is the first time for this brand-new (cloned) database, you get the standard disclaimer-dialog:

Select [Launch browser], then, create some J-Random node named 'Foo' or whatevs. You, thankfully, get back the 'Could not create token'-message. You've verified your database is in read-only mode.

So, now you can share your database, as it's a clone of your original database (I leave data-masking for you to figure out), and it's in read-only mode.

Share Cloned Read-Only Database

But how do you share your database?

Go to the Connection-tab for your database and there it contains the connection information, including username and password as well as the connection URI or the REST URL for accessing your database.

BOOM-BAM, BABY! You now have provided a public-access read-only copy of your database! And, as it's provided by grapheneDB.com DaaS, either by AWS or Azure (your choice; grapheneDB allows either), it is demoable anywhere you have internet access.

You are in business! It's up to you how you wish to present the data (transformed, masked, or straight up) or to scale the data (a part of your database or all of it) as your public-access database.

Now: go to! Go to!

Backup Plan...9 from Outer Space

How to make backups of your neo4j database on GrapheneDB DaaS

If you have a professional plan with GrapheneDB DaaS, then backups, for you, are automated. At the lowest professional tier, backups are only retained for a week, so continue to read the rest of this article for methods for saving data longer-term.

For the hobby-edition owners you get the following message:

Does this mean you are unable to make backups of your database? It appears so, at the face of it, but actually, no, you can make backups, and you can do this daily. The effort is on your side, but when it comes to the case that you've corrupted your database with a dirty load (as I have), these backups are an essential part of restoring integrity to your stored data.

So, let's do this.

How?

You export your database, and then you find some cloud-service to manage these daily exports, or, again, you take on the management of these exports. These exported copies of your database are, ipso facto, your daily backups.

To export your database, go to the Admin-tab and go to the "Export Database"-section. 

Select "Export Database," read the warning about database-stoppage, and select "Export Database," again.

Then, you're moved onto the next modal dialog – "Export Ready" – which offers a download of a snapshot of your database. Select "Download."

Boom! A snapshot of your database is now local. As you see at the bottom of your web-browser in your task-status bar:

Now that you have the snapshot, select "Close" on the modal. Officially, you're 'done.'

Now, what you do with it is up to you. You can move it to a hard medium (e.g.: DVD-RW, or shared network drive), or you can upload it to a cloud service, such as AWS S3 or your google drive. Your choice.

Backup management plan

Of course, you can save every daily backup forever, but what works best? From experience, the older the data gets, the less interesting it gets, so what I do is save the dailies for a week, then, beyond that, I save the first of every prior week up to a month (that is: I delete every non-monday backup older than a week), I save every first of month older than two months, then I save the first of the year. Usually, one finds a mistake the same day one makes it, so the rollback is one, two, or three days. Usually. Everything beyond immediacy is just paranoia, in my experience, so, out of paranoia I save data every month for a year and beyond that, every year. Once, twice or thrice, I've had to look in the archives older than a year to try to restore something I've lost. Oftentimes, the trouble of restoration is so great, I always say 'eh, whatevs,' anyway.

But, you know: saving older data, because reasons, and because everybody does it, so it Must Be Vital for my Health and Well-Being(tm).

As a point of reference the GrapheneDB DaaS saves one week of daily backups. Anything older than a week is discarded.

Nice to see some realists in the technology sector for a change.

Upgrading the Graph Database using ETL

Upgrading to Neo4J-2.3.2 database. ETL-in-action

We have the following etl.perl script:

geophf:1HaskellADay geophf$ cat ~/bin/etl.perl 

#!/usr/bin/perl

$junque = <>;

while(<>) {

   chomp;

   $day = $_;

   $one = <>;

   chomp $one;

   $two = <>;

   chomp $two;

   $three = <>;

   chomp $three;

   print `jsoner.sh $day "$one" "$two" "$three"`

}

which calls this shell script:

geophf:1HaskellADay geophf$ cat ~/bin/jsoner.sh 

#!/bin/bash

JSON=`jsonify $1 "$2" "$3" "$4"`

# echo $JSON

jsoneronious.sh $CYPHERDB_USER $CYPHERDB_PASSWD $CYPHERDB_URL "$JSON" $1

# echo my curl command has $CYPHERDB_USER:$CYPHERDB_PASSWD and $CYPHERDB_URL

echo

echo

echo Saved $1 top 5s to GrapheneDB

We've created a new GrapheneDB DaaS as (initialization-step):

See? It's really empty (verification-step):

Okay, let's update the access key-set with our new database name, then let's do this.

[updates database access key-set]

Then we make sure our top5s data source is up-to-date: http://lpaste.net/raw/4714982275408723968 by checking the most-recent (last) date:

2016-02-12

Mkt_Cap:JPM,WFC,PTR,GE,ATVI|LMCB,MOG.B,RAI,LPLA

Price:GRPN,ICPT,TCK,AXL,AVP|MOG.B,LPLA,JW.B,LMCB,P

Volume:BAC,GE,CSCO,CHK,QQQ,FCX,ATVI,AAPL,P,C

Okay, let's run this thing!

geophf:1HaskellADay geophf$ etl.perl Seer/data/top5s.csv 

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current

                                 Dload  Upload   Total   Spent    Left  Speed

100  2801  100   125  100  2676    557  11935 --:--:-- --:--:-- --:--:-- 11946

HTTP/1.1 100 Continue

HTTP/1.1 200 OK

Server: nginx

Date: Sun, 14 Feb 2016 06:33:58 GMT

Content-Type: application/json

Content-Length: 125

Connection: keep-alive

Access-Control-Allow-Origin: *

{"results":[{"columns":[],"data":[]},{"columns":[],"data":[]},{"columns":[],"data":[]},{"columns":[],"data":[]}],"errors":[]}

Saved 2016-02-12 top 5s to GrapheneDB

I didn't time it, but it took less than a minute to upload nine months of top5s data. Let's check our results:

Okay, great; let's do a specific query against $NFLX, on which I'm currently writing a case-study.

One last check. How much space did we take up in the new database? How much space do we have still available to us? Both for nodes (not so much a worry) and for storage:

As you can see, 9 months of data doesn't even scratch the surface of the developers' edition of the database. We have, oh, 500 or so years-worth of data before we need to start looking at other options. Sweetness.

BAM! The old database is therefore ready for decommissioning. We just have to remember now before we do the scrape to do a backup manually and confirm the new database is receiving new top5s information as we scrape it.

Fixing the Data Fix: Restoring from Backup

Data fix, part II, or: how to fix the data fix

So, yesterday, we looked at doing a hot data fix, and it ... worked, or it seemed to, but then, after adding the new day's data, instead of getting something like this:

this happened:

Oh, no! Now we have a data corruption, where stocks from the previous day and the current day are intermingled (the second graph-diagram) under one set of MKT_CAP, PRICE, and VOLUME headings instead of being separated into their respective days (as the first graph-diagram shows).

What to do? We have basically two options:

1. tease apart all the data relations for those two days, adding in the correct headings for each day and then partitioning the stocks to their respective days from the saved queries stored in the log files.

... you do save your log files don't you?

Basically, this is another hot fix. Possible? Yes. Doable? Yes. Risky?

... well: yes.

2. Or, we could make sure the data-load cypher queries are now correct offline (they are, I verified), blow away the current, corrupted database, restore from backup before the corrupt occurred and reload the corrected data from the stored cypher queries.

Basically, restore from backup.

Doable? Let's check. Do we have a backup? Well, thanks to grapheneDB.com, a DaaS ('Data as a Service'), we do, as they do a daily backup and save a week's worth of them for you under their professional plan. But before we start the restore, let's just be sure and save off our database as an export. You know: just in case we lose everything.

Exporting a database

How do we do that?

Simple. We go to the admin page which has the Export Database-option:

Then we select that option:

A database with six-month's worth of top 5s – under 2,000 nodes and under 2,000 relations – takes no time to prepare for export.  And then, we download that export:

Restoring from Backup

Now that we have a local export of our database, let's restore from out backup. 

The corruption occurred on 2016-01-05, so we use the 2016-01-04 backup:

... and accept the warning that we're about to wipe our database, because, yes, we want to eliminate the data corruption I introduced:

Then grapheneDB.com tells you, that, yes, your database is restored from backup:

Which is all well and good, but I'll do due-diligence, myself and confirm that the database is restored, firstly, and to the state before the corruption occurred.

Yup!

(Actually, I checked further into the database, and you should, too, to the level of your assurance, but this is not in the scope of this article.)

Data Correction from the point of the Restore

Okay, database restored to it's old state. WHEW! So we're back at ground zero, as it were, pre-2016. Now, let's reload our corrected 2016 data. To do that, I wrote a little utility that converts stock Top5s to Cypher-queries in JSON, called 'jsoner.sh.' I save off the top 5s to a daily record, the format of which is, e.g., this:

date>kind:Leaders|Losers

2015-12-31

Mkt_Cap:EPD,LBTYK,WMB,AAPL,GOOG|GOOGL,MSFT,AMZN

Price:NM-G,SWN,WPX,BCOM,LTRPB|AXON,NK,TXMD

Volume:GE,BAC,AAPL,KMI,QQQ,SPY,MSFT,SIRI,SUNE,FCX

2016-01-04

Mkt_Cap:EPD,BXLT,WFC-L,AMZN,GOOGL|GOOG,BABA,WFC

Price:SUNE,EPE,CHK,QUNR,AXON|RARE,NK,TEAM

Volume:SPY,BP,SAN,BHP,RIG,RDS.A,EEM,RIO

2016-01-05

Mkt_Cap:LLY,BABA,WMT,NTT,CHL|TOT,RDS.B,PTR,SAN

Price:SWHC,RGR,SHI,FLIR,GPRO|STRZB,CHK-D,BCOM,XLRN,EPE

Volume:ARRS,SPY,QQQ,XIV,SUNE,GPRO,SWHC,FCS,GDX

jsoner.sh scans the above and converts them to properly-formatted Cypher queries, ensuring the stocks fall into the the right heading for the appropriate day. So, let's run jsoner on 2016-01-04 and re-enter into the database the first corrected day's data:

Good. Let's verify those Top 5s are in and under the correct day:

And, you see at the bottom of the screen, I visually verified that the volume category is also now correct with the its appropriate date. It was incorrect before, showing 2016-01-05, and this was the root cause that lead to the data corruption before. Now we see that it is correct. YAY!

Okay, now let's reenter the corrected second day's data, that is the data for 2015-01-05 now:

And, we verify that the new data are in place, properly factored into the right days:

And so they are.

Database fully restored from a backup, ready, again, to receive the latest daily Top5s securities from the stock market.

We are back in business. YAY!