"How much $ETH do I have?" a Cryptocurrency pop-quiz

Mappers and Reducers.

Every computation can be expressed as a combination of mappers and reducers, and Big Data™️ leveraged this in a big way, because if you can express everything as mappers and reducers, you can parallelize on each mapper and reducer. 

The functional programming community further shows that every mapping function is a reducer.

map f = reduce (concat . f) []

All computations are reducers.

Therefore: your 1 instruction set computers (OISC)

And if you only need 1 instruction, you need 0 instructions (ZISC)

Okay, let's look at a mapping. We map from

to

with the following formula

map ((1),(k "Cryptocurrency"),(3),(4))

SIMPLE! We don't even need to write a program to do this transformation, as simple spreadsheet formula handle this directly. 

But how do we then reduce the spreadsheet of multiple crypto across multiple blockchains to 'just the crypto, please'-spreadsheet, something like this:

Let's review three approaches to this desired output.

1. One approach is to work within the spreadsheet, itself, using its functions, such as sumif() or dsum().

The problem I have with this approach is that you're using the spreadsheet as a database, and you have to learn these functions, which is learning a new language, really. 

2. Next, you can write a problem in, e.g. Rust, to take on the mappings and reductions.

I'm digging a modular approach here, where you code individual mappers and reducers, and you can plug-n-parallelize to your heart's content. 

3. As I've already codified my Blockaverse portfolio as an ontology, I simply query the ontology and receive a graph as a result.

Dude. 😎

Cryptocurrency pop-quiz

Choose one of the approaches above, or, forge your own path and create your own, then solve the problem of mapping a cryptocurrency portofolio and reducing it to tokens and their USD amounts across the Blockaverse.

DOIT! TOIT! 

• (answer)

Cryptocurrency pop-quiz: archive data into git

Those are many preexisting files to reduce to one file.

It'd be a shame to lose all that rich just-waiting-to-be-analyzed data.

I WONDER if'n there were a way to archive all those data, efficaciously?

#cryptocurrency pop-quiz

Write a program that archives those files into git.

• (answer)

What is HBase for?

Is HBase a good, flexible environment where you can do any kind of exploratory data mining that you need to do? Well, I can answer the question by the way we did it. It may not apply to your scenario, but then again, I don't know what your scenario is.

Have one scenario, was our scenario.

Let me explain.

HBase is wonderful, blazingly fast, and very specific. So, we store the stock market ('we' did, when I worked on that project, it's been a year since then, so, yes: I'm old). We stored it on GreenPlum at first. Why? Because we had to be able to handle any kind of query from the users, because that's what they said they want, and the users are always right.

That's the flexibility approach. What does that buy you? A limit of 200k-rows returned, max, and four hours per query with a maximum of 16 queries going on at any time.

We had 1600 users, so, yeah, that worked ... greeeeeeeat!

Or not.

So, we did studies, and we found that 85% of the queries were around a very specific query: the order lifecycle. That is, for any stock order (buy or sell), there was an order number for it, and that order number carried through all transactions for that order (placing the order, consolidating the order, sending it from a brokerage to the exchange, then executing the order). It would take hours to get 1 order out of GreenPlum ... because we'd get 6 billion orders every day, and the query to GreenPlum would reconstruct the order in the SQL. Ugh.

So, that's the 85%-rule. So, we simply took that one query, and built our HBase database around that. With order id being part of the index, bam, you got your order, any order, back in seconds.

Great.

But. A year later, people said: oh, I want to do research, I want to scan by date.

Hm. Problem. Date is not in the key, it's in the data, so to get date, you have to know the order (they don't) or your have to do a full table scan, 6 billion rows per day, 5 years of data.

Impossible. ... that is, now we're doing prototypes with Pegasus and HortonWorks by creating new indices on dates, but it took a month to do this prototype on a month's worth of data, and the database DOUBLED it size to accommodate this new index.

The date queries now go blazingly fast, but we're still undecided as to whether we want to bite the bullet on the agony of doubling our cluster size, our database size, just so somebody, if they want, can query by date.

Do you see what I'm saying here? HBase is NOT a general-purpose tool. It solves indexed data problems and it solves them blazingly fast. You start doing general purpose query and start having to scan values, then you may as well pack up and go home... OR create a new database where your sought values are now part of the indexed sets. So, yes, you can do that, but there's a cost in time (prototyping to ensure you're getting what you need, and then in creating the new database from the old database ... row by row) and in space, because now you have a new HBase database sharing the space with your old HBase database. And they are going to share space, even if just for a while, because if the new one blows up, you have to go back to the old, working one, so ... doubling your cost is the least expense you can hope for.

My experience: don't do exploratory querying against HBase outside the indices, that's not what HBase is for. HBase relieves agony that you had been having against the massive amount of data that you have and the set of VERY standardized queries you go against those data. 

Somebody always says: oh, that makes my life so much easier! So, can I do this one time thing that I'll never do again, but I'm just curious, and I don't care that it's an extreme boundary case that nobody cares about, I absolutely NEED this query because reasons.

Yeah. Be firm. This is what this database does. This is what this database DOES NOT do. If you want to do exploratory data mining, give me a start key and and end key and I'll give you a block of data to play with, otherwise the door is over there and here's a quarter to call somebody who cares.

My experience. HBase is awesome for what it is for. HBase sucks for things that it's not for, so don't use it for what it's not. Use it for what it is, and then trumpet your successes, harp on them, because people forget that they couldn't even think about exploratory data mining before you had the HBase database giving them the necessary answers first and in a timely fashion.

Fixing the Data Fix: Restoring from Backup

Data fix, part II, or: how to fix the data fix

So, yesterday, we looked at doing a hot data fix, and it ... worked, or it seemed to, but then, after adding the new day's data, instead of getting something like this:

this happened:

Oh, no! Now we have a data corruption, where stocks from the previous day and the current day are intermingled (the second graph-diagram) under one set of MKT_CAP, PRICE, and VOLUME headings instead of being separated into their respective days (as the first graph-diagram shows).

What to do? We have basically two options:

1. tease apart all the data relations for those two days, adding in the correct headings for each day and then partitioning the stocks to their respective days from the saved queries stored in the log files.

... you do save your log files don't you?

Basically, this is another hot fix. Possible? Yes. Doable? Yes. Risky?

... well: yes.

2. Or, we could make sure the data-load cypher queries are now correct offline (they are, I verified), blow away the current, corrupted database, restore from backup before the corrupt occurred and reload the corrected data from the stored cypher queries.

Basically, restore from backup.

Doable? Let's check. Do we have a backup? Well, thanks to grapheneDB.com, a DaaS ('Data as a Service'), we do, as they do a daily backup and save a week's worth of them for you under their professional plan. But before we start the restore, let's just be sure and save off our database as an export. You know: just in case we lose everything.

Exporting a database

How do we do that?

Simple. We go to the admin page which has the Export Database-option:

Then we select that option:

A database with six-month's worth of top 5s – under 2,000 nodes and under 2,000 relations – takes no time to prepare for export.  And then, we download that export:

Restoring from Backup

Now that we have a local export of our database, let's restore from out backup. 

The corruption occurred on 2016-01-05, so we use the 2016-01-04 backup:

... and accept the warning that we're about to wipe our database, because, yes, we want to eliminate the data corruption I introduced:

Then grapheneDB.com tells you, that, yes, your database is restored from backup:

Which is all well and good, but I'll do due-diligence, myself and confirm that the database is restored, firstly, and to the state before the corruption occurred.

Yup!

(Actually, I checked further into the database, and you should, too, to the level of your assurance, but this is not in the scope of this article.)

Data Correction from the point of the Restore

Okay, database restored to it's old state. WHEW! So we're back at ground zero, as it were, pre-2016. Now, let's reload our corrected 2016 data. To do that, I wrote a little utility that converts stock Top5s to Cypher-queries in JSON, called 'jsoner.sh.' I save off the top 5s to a daily record, the format of which is, e.g., this:

date>kind:Leaders|Losers

2015-12-31

Mkt_Cap:EPD,LBTYK,WMB,AAPL,GOOG|GOOGL,MSFT,AMZN

Price:NM-G,SWN,WPX,BCOM,LTRPB|AXON,NK,TXMD

Volume:GE,BAC,AAPL,KMI,QQQ,SPY,MSFT,SIRI,SUNE,FCX

2016-01-04

Mkt_Cap:EPD,BXLT,WFC-L,AMZN,GOOGL|GOOG,BABA,WFC

Price:SUNE,EPE,CHK,QUNR,AXON|RARE,NK,TEAM

Volume:SPY,BP,SAN,BHP,RIG,RDS.A,EEM,RIO

2016-01-05

Mkt_Cap:LLY,BABA,WMT,NTT,CHL|TOT,RDS.B,PTR,SAN

Price:SWHC,RGR,SHI,FLIR,GPRO|STRZB,CHK-D,BCOM,XLRN,EPE

Volume:ARRS,SPY,QQQ,XIV,SUNE,GPRO,SWHC,FCS,GDX

jsoner.sh scans the above and converts them to properly-formatted Cypher queries, ensuring the stocks fall into the the right heading for the appropriate day. So, let's run jsoner on 2016-01-04 and re-enter into the database the first corrected day's data:

Good. Let's verify those Top 5s are in and under the correct day:

And, you see at the bottom of the screen, I visually verified that the volume category is also now correct with the its appropriate date. It was incorrect before, showing 2016-01-05, and this was the root cause that lead to the data corruption before. Now we see that it is correct. YAY!

Okay, now let's reenter the corrected second day's data, that is the data for 2015-01-05 now:

And, we verify that the new data are in place, properly factored into the right days:

And so they are.

Database fully restored from a backup, ready, again, to receive the latest daily Top5s securities from the stock market.

We are back in business. YAY!

Data-fixin' down-low on the Production Data

You have this:

Note that the date is wrong: these Top5s stocks should be for 2016-01-04, yesterday, not today! AND we need to fix this by COB today, so our entire database does not become corrupted.

What to do?

So, we have three options: 

1. revert the database from the daily save. Safe. 

2. Remove that date and all its child nodes, ugh, and not so safe (children are children of every day). 

3. Or, reset that date.

Let's do 3. for a change: the hot data fix.

match (d:Day { month: 1})

set d.date = "2016-01-04"

set d.day = 4

return d

(n.b.: the above query works ONLY because there is only one trading day this new month (and year))

Fixed! Hot data fixes with Graph databases can be painless and easy, and you have visual proof your fix worked, too!

Post Script, 2016-01-05

Uh, oh! After adding next day we have this!

Data Corruption! Two days are sharing overlapping market caps, prices, and volumes! Root cause? I corrected the Day-node only, but the three child nodes (PRICE, VOLUME, MKT_CAP) have a date attribute that I did not correct in the data correction above! Oh, no! Data corruption!

What do we do? Besides run around in circles in a panic for a while?

Well, we restore from backup before this mess started and then re-add each days data using the correctly supplied dates, ... as we should have done in the first case instead of all this super-cool 'hot data fix'-approach. That's what we should have done.

Done. Corrected. VOLUME, PRICE, and MKT_CAP child nodes are now properly separated, and the stocks now fall under their respective day's categories.

Restoring from backup database with corrected data blog post forthcoming.

Executive Summary

Company Information

Logical Graphs LLC works with Big Data in various domains, including the stock market and exchanges, person-relation data sets, and transactional data. We analyze these data, find patterns, clusters, and relations, and present these results to our customers who then run campaigns, examine the markets (primarily for anomaly detection or non-conformance).

Products/Services

We use a NoSQL-approach to data collection and analysis. We have used HADOOP/HIVE and are currently focused on graph technologies, using neo4j as our graph database.

Our customers are both Government and Commercial clients that are realizing that they have Big Data, yes, but also a Huge data problem: that problem is that seeing relations and patterns in the data are hard or require experts, but seeing those patterns is vital to the execution of their mission or the effective and efficient running of their campaigns.

We, at Logical Graphs LLC, can help here. We have experience in grasping what the problems are with and in the currently existing data sets. We remodel the data in ways that speak the the problems needing to be solved at hand. We graph these data, exposing heretofore hidden relationships and then present the results that tells the story the data are trying to tell.

Using the above methodology:

• With requirements-as-graphs, we helped our customers rein in over-budget and past-due projects, focusing in on the high-impact requirements to deliver the most bang for the buck
  
• With personal-relations-as-graphs, we helped our customers target impact players, focus in on key issues, and run effective campaigns to influence policy and decisions
  
• With the historical data of the markets, we have identified interesting areas of the market which lend themselves to further data analysis
  
• With the Regressive Imagery Dictionary, we model the semantics and intents of documents, extracting the underlying meanings
  

Of course, if there's no report on the data and their findings, it's of no general use. Data analysis, if not presented or summarized well, can lead to no useful action being taken.

Report generation and synthetic data generation are our speciality.

Summary

You have data problems, not in storing and managing the data: modern databases handle that quite well for you. But what does it all mean? And how do you get it presented to your senior management so you can put it into the hands of your users so that they can be effective in their jobs?

That's where Logical Graphs LLC comes in. Let us present your data to you in ways you've never seen it before. Let us open up new opportunities for you that your data have hidden inside structures and relations waiting to be discovered.